Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents

🔼 Agent S2 is a compositional framework that uses both generalist and specialist modules to handle complex computer tasks that require long-term planning. The figure shows the system’s architecture. The Manager (M) handles high-level planning, breaking down complex tasks into smaller sub-goals. The Worker (W) executes these sub-goals, using specialist grounding experts (MoG) to accurately locate and interact with GUI elements. The interaction between these modules is dynamic and iterative, allowing Agent S2 to adapt to changing circumstances and unexpected events.

read the caption

Figure 2: The Agent S2 framework. It composes generalist planning modules, Manager M𝑀Mitalic_M and Worker W𝑊Witalic_W, with specialist grounding experts to complete complex, long-horizon computer use tasks. Please refer to Section 3 for a detailed explanation.

🔼 This figure illustrates the core difference between reactive and proactive planning approaches in the context of hierarchical planning for computer use agents. Reactive planning rigidly follows a pre-defined plan, only adjusting if a subtask fails. In contrast, proactive planning dynamically updates the remaining plan after each completed subtask, allowing for adaptation to evolving observations and increased robustness in response to unforeseen circumstances.

read the caption

Figure 3: Comparison between Reactive and Proactive Planning. Proactive planning re-evaluates and updates the remainder of the plan after every subtask, while reactive planning adheres to a fixed plan and only revises it in response to subtask failures.

🔼 Agent S2 initially tries to select a paragraph using the Visual Grounding Expert, but this attempt fails. It then self-corrects by switching to the Textual Grounding Expert, which successfully selects the desired span of text. Once this subgoal is achieved, Agent S2 proactively replan its actions, updating its internal state to reflect the successful completion of the subgoal and then proceeds to the next subgoal in its plan.

read the caption

Figure 4: Agent S2 attempts to use Visual Grounding Expert to select a paragraph, then self-corrects and uses Textual Grounding Expert for span selection. After completing the subgoal, it replans from the new state and starts working on a new subgoal.

🔼 This ablation study investigates the individual and combined effects of Mixture of Grounding (MoG) and Proactive Hierarchical Planning (PHP) on the success rate of Agent S2. The graph displays success rates for different configurations: Agent S2 with both MoG and PHP, Agent S2 without MoG, Agent S2 without PHP, and several baseline agents (UI-TARS-72B-DPO, UI-TARS-7B-DPO, UGround-V1-7B, and Claude-3.7-Sonnet). The results show the performance gains achieved by incorporating both MoG and PHP, highlighting their importance for improved performance on 15-step and 50-step tasks.

read the caption

Figure 5: Ablation study on Mixture of Grounding (MoG) and Proactive Hierarchical Planning (PHP).

🔼 This figure presents a bar chart comparing the 15-step success rates of Agent S2 using different visual grounding models. The results show that smaller, open-source models achieved higher success rates compared to larger, closed-source reasoning models. This highlights the potential advantage of leveraging smaller, more specialized models for specific tasks within a larger, compositional agent architecture.

read the caption

Figure 6: 15-step performance of Agent S2 with different visual grounding models. Small open-sourced models can outperform large close-sourced reasoning models.

🔼 This figure shows the breakdown of reasons why Agent S2, a computer-use agent, improves its performance when given more time steps (from 15 to 50). The analysis focuses on successful cases and identifies four key contributing factors: Adaptive Navigation (exploring different ways to interact), Adaptive Interaction (trying different methods to achieve a goal), Backward Correction (correcting previous mistakes), and Task Complexity (inherent difficulty of tasks). The percentages indicate the contribution of each factor to the overall improvement in success rate.

read the caption

Figure 7: Contributing factors of 15-step to 50-step scaling success.

🔼 This figure shows a breakdown of the reasons why Agent S2 failed on a subset of tasks from the OSWorld benchmark. It categorizes failures into several key areas, such as problems with the overall plan, issues with grounding (locating specific UI elements), difficulties interacting with elements, problems with navigation within the application, and tasks that were inherently infeasible to complete. The size of each category’s slice reflects the proportion of failures attributed to that particular cause. This helps to identify and analyze the main challenges faced by the agent.

read the caption

Figure 8: Failure categories on a subset of the OSWorld benchmark.

🔼 Agent S2 is tasked with adding strikethrough formatting to the last paragraph of a document. The figure shows the agent’s step-by-step process using textual grounding. Initially, it attempts visual grounding but self-corrects, demonstrating adaptive behavior. It then leverages the textual grounding expert to precisely select the target words before applying the formatting. This highlights the system’s ability to accurately identify and manipulate specific text within a document, showcasing its advanced grounding capabilities. The precise word selection is key, avoiding the limitations of selecting larger text blocks, demonstrating sophisticated text understanding.

read the caption

Figure 9: Agent S2 demonstrates textual grounding. The task instruction is: I think the last paragraph is redundant so I want to add strike-through on words in the last paragraph. Can you do this for me? Agent S2 intelligently utilizes the textual grounding expert to perform a precise word-level selection of text.

🔼 Agent S2 is shown using structural grounding to add a new column named ‘Profit’ to a spreadsheet. The goal is to calculate profit for each week by subtracting the COGS (cost of goods sold) from the Sales figures. The figure visually shows the steps involved, highlighting how Agent S2 interacts with the spreadsheet cells at a low level. This involves using a specialized ‘structural grounding expert’ that understands spreadsheet structure to perform fine-grained actions within cells, such as entering formulas and dragging to fill in values. The adaptive nature of Agent S2 is apparent, dynamically adjusting its actions based on the evolving state of the spreadsheet. This illustrates Agent S2’s ability to handle detailed, low-level operations within specific applications.

read the caption

Figure 10: Agent S2 demonstrates structural grounding. The task instruction is: Add a new column named “Profit” and calculate the profit for each week by subtracting “COGS” from “Sales” in that column. Agent S2 adaptively interacts with the “Profit” column and chooses to use the structural grounding expert to perform cell-level operations.

🔼 Agent S2 receives the instruction to disable the ‘Dim screen when inactive’ setting in the system settings. However, this exact setting name is not present in the user interface. The figure shows how Agent S2, using its proactive hierarchical planning, dynamically adjusts its plan. Initially, it attempts to locate the setting directly. Upon failure to find the verbatim setting, it re-evaluates the situation, exploring related settings (like screen brightness or power saving modes). It revises its plan, identifying alternative options that achieve the user’s intent. This process of replanning and exploration, illustrated through screenshots of the user interface, demonstrates Agent S2’s adaptive behavior and problem-solving capabilities in uncertain environments.

read the caption

Figure 11: Agent S2 demonstrates replanning. The task instruction is: Could you set the ‘Dim screen when inactive’ to off in settings? Even though the ‘Dim screen when inactive’ option is not found in verbatim, Agent S2 proactively replans and reasons to find the correct setting.

🔼 Agent S2 successfully completes a complex task on the WindowsAgentArena benchmark. The task involves creating a desktop shortcut for a folder, requiring multiple steps and interaction with the Windows GUI. The figure showcases Agent S2’s ability to dynamically adjust its plan, refining its actions based on the evolving state of the environment and feedback from the interface. This highlights the effectiveness of the Proactive Hierarchical Planning component of the Agent S2 framework.

read the caption

Figure 12: Agent S2 on the WindowsAgentArena environment. The task instruction is: Create a shortcut on the Desktop for the folder named ”Projects” that is located in the Documents folder. Name the shortcut ”Projects - Shortcut”. Through consistent replanning, Agent S2 is able to dynamically update its plan and revise its current subtask in more detail.

🔼 Agent S2 successfully navigates the AndroidWorld environment to create a new contact. The figure displays a sequence of screenshots showing the agent’s interactions, starting with opening the Contacts app and culminating in entering all requested contact details (First Name, Last Name, Phone number, and Phone Label). This demonstrates Agent S2’s ability to handle touch, typing, and dropdown menu selection actions within a mobile GUI.

read the caption

Figure 13: Agent S2 on the AndroidWorld mobile environment. It utilizes open, touch, and type interactions to complete the instruction “Go to the new contact screen and enter the following details: First Name: Grace, Last Name: Taylor, Phone: 799-802-XXXX, Phone Label: Work”.

🔼 This table presents a breakdown of Agent S2’s performance on the OSWorld 50-step benchmark, categorized by task type (OS, Daily, Office, Professional, Workflow). Success rates are shown, illustrating the model’s effectiveness across various task domains. The results are further broken down to show the impact of using different large language models (LLMs) as the Manager and Worker components within the Agent S2 framework.

read the caption

Table 2: Categorized Success Rate (%) of Agent S2 on the OSWorld 50-step evaluation. We report results with various MLLMs as Manager and Worker.

🔼 This table presents the success rates of different computer use agents on the WindowsAgentArena benchmark, focusing on the performance within 15 steps. It highlights Agent S2’s state-of-the-art (SOTA) achievement by comparing its performance against baselines like Agent S and NAVI. A key differentiator is that Agent S2 uniquely relies only on screenshots as input, unlike Agent S and NAVI, which also utilize accessibility trees. This comparison underscores Agent S2’s superior performance in achieving the SOTA despite its use of less information.

read the caption

Table 3: Success Rate (%) on the WindowsAgentArena test set (within 15 steps). Note that both Agent S and NAVI use screenshots and accessibility trees, while our agent only takes screenshots as the input. Agent S2 sets new SOTA on WindowsAgentArena.

🔼 This table presents the results of Agent S2’s performance on the AndroidWorld benchmark, specifically focusing on tasks involving smartphone use. It compares Agent S2’s success rate to other state-of-the-art methods.

read the caption

Table 4: Agent S2 results on AndroidWorld for smartphone use.

🔼 This table presents a detailed breakdown of Agent S2’s performance on the OSWorld benchmark’s 50-step evaluation. It shows success rates across different application domains (OS, GIMP, VS Code, Thunderbird, LibreOffice Writer, Calc, Impress, Chrome, VLC, and Multiapps). Importantly, it also demonstrates how performance varies depending on the specific large language models (LLMs) used as the ‘Manager’ and ‘Worker’ components within the Agent S2 framework.

read the caption

Table 5: Success rate (%) of Agent S2 on 50 step evaluation in OSWorld, divided by domains: OS, GIMP, VS Code, Thunderbird, LibreOffice Writer, LibreOffice Calc, LibreOffice Impress, Chrome, VLC, and Multiapps. We also report performance using different models as the Manager and Worker backbones.

🔼 This table presents the success rate of Agent S2, a computer use agent, on the OSWorld benchmark. The success rate is broken down by specific application domains (OS, GIMP, Code, Thunderbird, LibreOffice Writer, LibreOffice Calc, LibreOffice Impress, Chrome, VLC, Multiapps), providing a granular view of performance across different task types. The results are shown for a 15-step evaluation, and the table also includes comparative results using different models (Claude-3.5-Sonnet, Claude-3.7-Sonnet, GPT-40) as the Manager and Worker components within the Agent S2 framework.

read the caption

Table 6: Success rate (%) of Agent S2 on 15 step evaluation in OSWorld, divided by domains. We also report performance using different models as the Manager and Worker backbones.

🔼 This table presents a detailed breakdown of Agent S2’s performance on the WindowsAgentArena benchmark. It shows the success rate achieved for each of 12 different applications, categorized by application type (browser, code editor, office suite, etc.). Agent S2, using the Claude-3.7-Sonnet model, is evaluated on tasks within these applications. The success rate is expressed as a percentage and provides a granular view of the agent’s capabilities across various software types.

read the caption

Table 7: Success rate (%) of Agent S2 with Claude-3.7-Sonnet on WindowsAgentArena, divided by apps: Chrome, Microsoft Edge, VS Code, Notepad, LibreOffice Calc, Settings, Windows Calc, Clock, Microsoft Paint, File Explorer, LibreOffice Writer, VLC Player.

🔼 This table details the actions Agent S2 can perform when interacting with a graphical user interface (GUI). For each action (e.g., click, type, scroll), it lists the action type, a description of its purpose, and the arguments required to specify the action’s parameters. This provides a comprehensive overview of Agent S2’s capabilities in manipulating GUI elements.

read the caption

Table 8: Agent S2 action space, descriptions, and arguments.