ShowUI: One Vision-Language-Action Model for GUI Visual Agent

🔼 Figure 2 presents a comparison of ShowUI with other GUI visual models. The left panel shows a scatter plot illustrating the relationship between model size, training dataset size, and zero-shot Screenspot grounding accuracy. ShowUI achieves state-of-the-art accuracy while being significantly more lightweight (2B parameters) and using a smaller training dataset (256K) compared to other models. The right panel demonstrates the impact of ShowUI’s UI-guided token selection. Compared to its base model (Qwen2-VL-2B), ShowUI reduces visual token redundancy by 33% during training, resulting in a 1.4x speedup.

read the caption

Figure 2: Left: Zero-shot Screenspot grounding comparison between ShowUI and other GUI visual models in terms of model size and training scale (area); ShowUI reaching state-of-the-art accuracy as well as the most lightweight model (2B) with a smaller training dataset (256K). Right: Building upon Qwen2-VL-2B [41], our UI-guided visual token selection reduces visual token redundancy by 33% during training, achieving a 1.4×1.4\times1.4 × speedup.

🔼 ShowUI is a vision-language-action model for GUI automation. It takes a user’s task query and an initial screenshot as input. ShowUI then iteratively executes actions, updating the screenshot after each action. This cycle continues until the task is complete. Key innovations include UI-Guided Visual Token Selection (efficiently processing screenshots by creating a UI-connected graph and randomly selecting tokens during training) and Interleaved Vision-Language-Action Streaming (handling past screenshots and actions to improve navigation).

read the caption

Figure 3: Illustration of ShowUI. Given a user task query, a pre-defined action space, and an initial screenshot as observation, ShowUI proceeds by executing the next action, updating the screenshot, and continuing in this cycle. Notably, ShowUI features the following key innovation designs: (i) UI-Guided Visual Token Selection, which processes the input screenshot to build UI patch-wise connected graph. During training, a random subset of tokens is selected within each component for efficient visual modeling (Sec. 2.1). (ii) Interleaved Vision-Language-Action Streaming to effectively handle past screenshots and actions, improving navigation performance. (Sec. 2.2).

🔼 The figure illustrates how UI-Guided Visual Token Selection works. It shows that the algorithm processes an input screenshot and groups similar pixels together, forming connected components. These components represent regions of the screenshot with similar visual features. This grouping is adaptive and depends on the screenshot’s content; areas with many similar pixels (e.g., background) are collapsed into fewer components, while areas with diverse pixels (e.g., interactive elements) are divided into more components. This method reduces the number of visual tokens needed for processing, improving the model’s efficiency without losing crucial information.

read the caption

(a) UI Connected Graph adaptively assigns connected components based on the informativeness of screenshots.

🔼 This figure compares two methods for visual token compression using a UI-connected graph. The UI-connected graph groups similar image patches together based on their RGB values. Patches of the same color belong to the same component and represent redundant information. The first method, token merging, combines all tokens within a component into a single token, potentially losing spatial information. The second method, token selection, randomly selects a subset of tokens from each component, preserving positional information. This illustration helps explain how the UI-connected graph addresses redundancy in high-resolution screenshots, and how the token selection method is superior to token merging for preserving positional information during the self-attention process.

read the caption

(b) Two representative token compression methods, where patches of the same color indicate the same component and are redundant to each other.

🔼 This figure illustrates the UI-Guided Visual Token Selection method. The left side shows how a screenshot is divided into a 28x28 patch grid, resulting in 1296 tokens. These tokens are then grouped into connected components based on their RGB values using a UI Connected Graph. Patches within the same component are considered redundant. The right side compares two approaches for using the UI Connected Graph: token merging (combining all tokens in a component, losing positional information) and token selection (randomly sampling tokens from each component, preserving positional information).

read the caption

Figure 4: Illustration of UI-guided Visual Tokens Selection. Left: Starting from an original screenshot (left) with a 28x28 patch grid (middle), resulting in 1296 tokens, the UI Connected Graph adaptively organizes the image into disjoint regions based on RGB values, allowing patches within the same region to be treated as redundant. Right: Comparison of two methods leveraging UI Connected Graph in visual token modeling: Token merging pools all tokens within one component, which loses individual positional information, while token selection, randomly sample part of tokens with each component, still retains their original position relationship.

🔼 The figure illustrates the UI-Guided Visual Token Selection method. It starts with a screenshot containing 1272 tokens (after a standard patching process). Using this method, these tokens are grouped into 781 components based on their visual similarity in RGB space. This grouping aims to identify and reduce redundant visual information in the screenshot, which is critical for improving model efficiency. Patches with the same RGB values are grouped into the same components, effectively reducing the computational cost of the self-attention blocks in the visual encoder.

read the caption

(a) 1272 tokens →→\rightarrow→ 781 components

🔼 This figure shows the result of applying UI-Guided Visual Token Selection to a screenshot. The screenshot initially contains 1272 tokens after standard patching. The UI-Guided Visual Token Selection method, described in section 2.1 of the paper, processes the screenshot and groups redundant image patches into connected components. After this process, the number of components is reduced to 359, illustrating the effectiveness of the method in reducing the number of visual tokens that the model needs to process.

read the caption

(b) 1272 tokens →→\rightarrow→ 359 components

🔼 This figure shows the result of applying UI-Guided Visual Token Selection to a screenshot from a mobile device. The original screenshot was initially divided into 1272 visual tokens. After processing with the UI-Guided Visual Token Selection method, these tokens were grouped into 265 connected components. This reduction in the number of tokens is a key aspect of the proposed method, significantly improving the efficiency of the model by reducing redundancy.

read the caption

(c) 1272 tokens →→\rightarrow→ 265 components

🔼 This figure shows a screenshot from a mobile device, specifically illustrating how the UI-Guided Visual Token Selection method reduces the number of tokens required to represent the image. The original screenshot, after patching, results in 1272 tokens. However, by utilizing the UI-connected graph, redundant patches are identified and grouped together, effectively reducing the total number of components (and thus tokens) to 175. This demonstrates the efficiency of the method in reducing computational cost by focusing on only essential visual information.

read the caption

(d) 1272 tokens →→\rightarrow→ 175 components

🔼 This figure shows a screenshot from a PC environment that was initially divided into 646 visual tokens using a 28x28 patch grid. The UI-Guided Visual Token Selection method then grouped these tokens into connected components based on their RGB values. The resulting UI Connected Graph reduced the number of components to 281, thus significantly reducing redundancy in the visual representation.

read the caption

(e) 646 tokens →→\rightarrow→ 281 components

🔼 This figure shows an example of how the UI-connected graph is constructed for a screenshot. The screenshot from a PC application was divided into patches, resulting in 646 tokens. Then, the UI-connected graph method grouped similar patches together into 230 components.

read the caption

(f) 646 tokens →→\rightarrow→ 230 components

🔼 This figure shows an example of how the UI-Connected Graph method constructs a visual representation from a screenshot. Specifically, it demonstrates the process for a web screenshot. Initially, the screenshot is divided into 1296 patches (tokens). The UI-Connected Graph algorithm then groups these patches into 740 connected components based on their RGB values. Patches with similar RGB values are grouped together, representing redundant information. This reduction in the number of components, from 1296 to 740, showcases the effectiveness of the UI-Guided Visual Token Selection method in reducing computational complexity by focusing on visually distinct regions of the screenshot.

read the caption

(g) 1296 tokens →→\rightarrow→ 740 components

🔼 This figure shows the result of applying UI-Guided Visual Token Selection to a web screenshot. Initially, the screenshot is divided into 1296 image patches (tokens). The algorithm then groups similar patches based on their RGB values into connected components. After this process, the screenshot is represented by 369 components, significantly reducing redundancy and computation.

read the caption

(h) 1296 tokens →→\rightarrow→ 369 components

🔼 Figure 5 demonstrates the UI-connected graph construction method used in the ShowUI model. The method leverages the inherent structure of UI screenshots to reduce computational cost by identifying and grouping redundant visual information. Each screenshot is divided into patches, represented as nodes in a graph. Nodes representing patches with similar RGB values are connected, forming connected components. This approach effectively models redundancy in screenshots. The figure shows examples from different device types: (a-d) Mobile, (e-f) PC, and (g-h) Web, highlighting how the connected component representation varies based on the visual structure of different interfaces.

read the caption

Figure 5: Illustration of our method constructs the UI-connected graph based on the informativeness of screenshots. (a–d) Mobile; (e–f) PC; (g–h) Web.

🔼 This figure illustrates the proposed Interleaved Vision-Language-Action (VLA) Streaming method. It highlights the significant length disparity between visual tokens from screenshots (around 1300) and textual tokens from queries or actions (less than 10). To address this imbalance and improve training efficiency, the method introduces two modes: Action-Visual Streaming, which handles the sequential nature of UI navigation tasks by incorporating visual history, and Action-Query Streaming, which is designed for UI grounding tasks and efficiently utilizes data through multi-turn interactions, similar to multi-turn dialogues.

read the caption

Figure 6: Illustration of Interleaved Vision-Text-Action Streaming. The visual tokens in screenshots are significantly longer (e.g., 1.3K) compared to queries or actions (e.g., fewer than 10). Thus, we introduce two modes: (a) Action-Visual Streaming for UI navigation and (b) Action-Query Streaming for UI grounding. These modes extend the concept of multi-turn dialogue and enable more efficient utilization of training data.

🔼 Figure 7 shows how the authors utilize GPT-4 to generate diverse queries for UI elements. Starting with the original, often simplistic, annotation (e.g., ‘message_ash’), GPT-4 is prompted with the visual context (screenshot and highlighted element) to produce three richer query types: Appearance (describing visual features), Spatial Relationship (locating the element relative to other UI elements), and Intention (describing the user’s likely goal for interacting with the element). This process significantly enriches the training data by adding more nuanced and descriptive labels beyond basic element names.

read the caption

Figure 7: We derive three types of query (appearance, spatial relationship, and intention) from raw annotation, assisted by GPT-4o.

🔼 This figure shows an example of how the UI-Guided Visual Token Selection method is applied to a weather application screenshot. The original annotation for this element is ‘visibility’. The caption breaks down the element’s visual characteristics (appearance: a rectangular box with 28 km in white text), its location within the UI (spatial: positioned below ‘WIND’ and next to ‘PRESSURE’), and the user’s likely intent when interacting with it (intention: check current fog or mist conditions). This detailed caption highlights the different ways the model can perceive and understand UI elements.

read the caption

(a) Example of Weather. Original: ‘visibility’; Appearance: “A rectangular box with 28 km in white text.”; Spatial: “Positioned below ‘WIND’ and next to ‘PRESSURE’.”; Intention: “Check current fog or mist conditions.”

🔼 This figure shows an example from the Stocks dataset used in the paper. The original annotation describes the UI element as ‘Share option-health insurance.’ The image itself shows three vertical dots in a dark background. Spatially, this icon is located to the right of the health insurance headline. The intended user action for this element is to click it to share the health insurance article. This example highlights the dataset’s focus on detailed visual, spatial, and functional descriptions of UI elements.

read the caption

(b) Example of Stocks. Original: ‘Share option-health insurance’; Appearance: “Three vertical dots icon on a dark background.”; Spatial: “Located to the right of the health insurance headline.”; Intention: “Click to share the health insurance article.”

🔼 This figure shows an example from the WeChat app. The original annotation for this UI element is ’expand_profile’. A more detailed description of the element’s visual appearance is provided: it’s a rounded gray button with a person icon. The spatial location is described as being in the top-left corner of the chat pane. Finally, the intended action or purpose of this button is to expand the contact’s profile view. This detailed caption highlights how visual and contextual information is annotated in the dataset used for training.

read the caption

(c) Example of WeChat. Original: ‘expand_profile’; Appearance: “A rounded gray button with a person icon.”; Spatial: “Located at the top-left corner of the chat pane.”; Intention: “Expand the contact’s profile view.”

🔼 This figure shows an example from the OmniAct dataset illustrating how diverse queries can be generated for a single UI element using GPT-4. The original annotation only states the UI element is a ‘Play’ button. Using GPT-4, the researchers extracted several types of descriptions that go beyond a simple name: (1) Appearance: describing the visual properties of the button such as color and shape (a white triangle within a black circle). (2) Spatial: explaining where the button is located on the screen (bottom left corner). (3) Intention: stating the purpose of the button (to play the video). These three descriptions offer different ways to interact with and understand the UI element, which are helpful for training a robust visual agent.

read the caption

(d) Example of VLC. Original: ‘Play’; Appearance: “White triangle icon within a black circular button.”; Spatial: “Located at the bottom left corner of the screen.”; Intention: “Click to play the video.”

🔼 This figure shows an example of how the UI-connected graph is constructed for a terminal screenshot. The image is broken down into patches, and patches with similar RGB values are grouped into connected components. The original annotation is ‘create_new_tab’, but the model also considers appearance (‘A small ‘+’ icon in a gray tab bar’), spatial location (‘Located at the far right of the tab bar’), and intended function (‘Open a new terminal tab’) to better understand the screenshot.

read the caption

(e) Example of Terminal. Original: ‘create_new_tab’; Appearance: “A small ’+’ icon in a gray tab bar.”; Spatial: “Located at the far right of the tab bar.”; Intention: “Open a new terminal tab.”

🔼 This figure shows an example from the OmniAct dataset, specifically a screenshot from a ‘Todo’ application. The caption details four aspects of the image to help illustrate how the model is trained to understand visual elements within GUI interfaces. The ‘Original’ label refers to the original annotation of this specific item, ‘view as list.’ The ‘Appearance’ description is a summary of the visual properties: a gray, vertical button with a box and lines icon. ‘Spatial’ gives the location relative to the rest of the UI, positioned at the top right beside the search bar. Finally, ‘Intention’ describes what action the button is associated with: switching to a list view of the task items.

read the caption

(f) Example of Todo. Original: ‘view as list’; Appearance: “A gray, vertical button with a box and lines icon.”; Spatial: “Positioned at the top right beside the search bar.”; Intention: “Switch to list view.”

🔼 Figure 8 shows examples of how the authors enhanced the OmniAct-Desktop dataset. The original dataset only provided basic labels for UI elements. To enrich this, the authors used GPT-4 to generate more descriptive queries for each UI element. These queries are categorized into three types: Appearance (describing visual characteristics), Spatial Relationships (describing the element’s location relative to other elements), and Intention (describing the user’s likely goal when interacting with the element). This augmentation resulted in a more comprehensive and diverse dataset for training their GUI visual agent.

read the caption

Figure 8: Illustration of how we augment the original OmniAct-Desktop annotations with diverse queries based on Appearance, Spatial Relationships, and Intention.

🔼 Table 4 presents a comparison of different models’ performance on the Mind2Web web navigation benchmark. The table shows the performance in terms of Element Accuracy (Ele.Acc), Operation F1-score (Op.F1), and Average Steps to Success (Step.SR). Models marked in gray require either HTML text inputs or access to the closed-source GPT-4V language model. ShowUI† denotes a variant of the ShowUI model that uses only action history, not visual history. Importantly, the table highlights that ShowUI achieves zero-shot performance comparable to SeeClick, which is a model that uses both pre-training and fine-tuning. This demonstrates the effectiveness of the ShowUI model.

read the caption

Table 4: Web Navigation on Mind2Web. The gray correspond to methods that required HTML text inputsor rely on close-source GPT-4V (marked with *). ShowUI ††\dagger† denotes our variant utilizing only action history. ShowUI’s zero-shot performance yield comparable score with SeeClick with pretrained and fine-tuning.

🔼 This table presents the results of online navigation experiments conducted using the MiniWob benchmark. Specifically, it shows the performance of the ShowUI model and its variant (ShowUI†), which uses only action history, on 35 tasks. The results are compared against the SeeClick model [11], a relevant benchmark for this task, showcasing the effectiveness of ShowUI in a dynamic online environment. The table likely includes metrics such as accuracy, success rate, or other relevant performance indicators.

read the caption

Table 5: Results on online navigation MiniWob on 35-tasks split following SeeClick [11]. ShowUI ††\dagger† denotes our variant utilizing only action history.

🔼 This figure compares three different methods for compressing visual tokens in a vision-language model: a baseline method with no compression, a token merging approach, and a token selection approach guided by a UI-connected graph. The comparison is based on three key metrics: the average length of visual tokens across all layers (#Vis.ctx), the training speedup achieved compared to the baseline method (Train.Speedup), and whether each method can be used during inference (Inference). The UI-connected graph method is shown to be superior, providing a substantial reduction in token length and training time without sacrificing inference capability.

read the caption

(g) Comparison between different visual tokens compression methods. ‘#Vis.ctx’ represents the avg. length of visual tokens across all layers. ‘Train.Speedup’ denotes the training efficiency improvement over the baseline. ‘Inference’ denotes whether this method is applicable at test time.

🔼 This ablation study explores the impact of inserting UI-guided token selection at different layers of the model. It compares the performance of inserting the method across all layers, early layers, late layers, and cross layers (alternating between inserted and non-inserted layers). The goal is to determine the optimal layer(s) for incorporating this technique to maximize performance.

read the caption

(h) Different insertion layers.

🔼 This ablation study investigates the impact of different token selection ratios on the model’s performance. The token selection ratio controls how many tokens are randomly selected within each UI-connected component during training. The figure likely shows the trade-off between reducing computational cost (by selecting fewer tokens) and maintaining sufficient model accuracy (by retaining enough tokens for the self-attention mechanism). Different ratios are tested, and their resulting accuracy on the Screenspot benchmark are compared.

read the caption

(i) Different selection ratio.

🔼 This figure presents ablation studies that analyze the impact of different design choices related to the UI-Guided Token Selection method. It explores the effects of various token compression methods (Token Merge, Random Token Selection, and UI-Graph guided Token Selection), the optimal layer for inserting the selection mechanism (early, late, cross-layer, or all layers), and the impact of different selection ratios on model performance and training speed. The results show the effectiveness of the proposed UI-guided token selection method compared to alternative techniques.

read the caption

Figure 9: Ablation studies of several design chocies regarding our UI-Guided Token Selection.

🔼 This figure displays the impact of using interleaved action-query streaming on the performance of a grounding task. The model was trained using 119,000 data points focused on grounding, and then evaluated on the Screenspot benchmark. The graph shows how the model’s performance changes over the course of training, comparing a one-turn action approach to a multi-turn approach. The multi-turn approach allows the model to process multiple queries and actions within a single training step, potentially improving efficiency and accuracy.

read the caption

Figure 10: Impact by Interleaved action-query streaming on Grounding task: trained with 119K grounding data, Eval with Screenspot.

🔼 This table presents the results of a zero-shot grounding experiment using the Screenspot benchmark dataset. It compares the performance of ShowUI, a lightweight vision-language-action model, against other state-of-the-art models. The comparison considers model size, training data size, and grounding accuracy across different device types (Web, Mobile, Desktop). The asterisk (*) indicates that the Omniparser model utilized GPT-4V. ShowUI-G represents a variant of ShowUI trained exclusively on grounding data, without using navigation data. The results show that ShowUI achieves strong performance with a small model size and relatively little training data.

read the caption

Table 2: Zero-shot grounding on Screenspot. * means Omniparser use GPT-4V. “Size” refers to model size. ShowUI-G: trained solely on grounding data, excluding navigation data. ShowUI, delivers strong grounding results with a lightweight model size and minimal training data.

🔼 Table 3 presents a comparison of different models’ performance on mobile navigation tasks from the AITW benchmark [36]. The table shows the accuracy of each model in terms of element accuracy and operational F1-score, as well as average steps to reach the goal in navigation tasks. Models marked with an asterisk (*) use either HTML inputs or rely on the closed-source GPT-4V language model. The entry for ShowUI† indicates the performance of a variant of the ShowUI model that does not use interleaved vision-action streaming, relying solely on action history.

read the caption

Table 3: Performance of Mobile Navigation [36], where gray color indicates these methods either using HTML inputs or rely on close-source GPT-4V (marked with *). ShowUI ††\dagger† denotes our variant without interleaved visual-action streaming, utilizing only action history.

🔼 This table presents the results of an ablation study, evaluating the impact of individual training datasets on the Screenspot benchmark’s zero-shot grounding performance. It shows the performance gains achieved by using different combinations of training datasets (AMEX, Web, OmniAct, and GUIAct) with various data augmentation methods. The data demonstrates the effects of each data source and the benefit of balanced data sampling strategies for improved model generalization.

read the caption

Table 6: Effect by individual training data on Screenspot.