JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mouse

🔼 This figure illustrates the difference between traditional VLA training methods and the proposed ActVLP approach. Traditional methods directly fine-tune a vision-language model on large-scale datasets using imitation learning to predict actions. In contrast, ActVLP is a three-stage training pipeline. Stage 1 post-trains the language model on text-based world knowledge using supervised fine-tuning with next-token prediction. Stage 2 further post-trains both the vision encoder and language model on multimodal datasets focusing on vision-language alignment and spatial grounding, again using supervised fine-tuning with next-token prediction. Finally, Stage 3 post-trains only the language model on multimodal instruction-following datasets via imitation learning. This multi-stage approach aims to enhance the model’s world knowledge, visual understanding, and spatial reasoning capabilities before relying solely on imitation learning for action prediction.

read the caption

Figure 2: Previous VLA methods usually directly use imitation learning to finetune original vision-language models on large-scale multi-domain decision-making datasets to predict the actions [25, 7]. Our ActVLP training pipeline includes three stages: 1) post-training language models on text-only world knowledge with next-token prediction supervised fine-tuning, 2) post-training both vision encoder and language models on multimodal vision-language alignment and spatial grounding datasets with next-token prediction supervised fine-tuning, and 3) post-training only language models on multi-modal instruction following datasets with imitation learning.

🔼 Figure 3 illustrates the diverse visual-language datasets used for post-training the model. A unified tokenizer allows the model to leverage these datasets, improving its performance on various vision-language tasks including question answering, image captioning, image/video question answering, and visual grounding (which incorporates both point and bounding box annotations). This multifaceted approach enhances the model’s decision-making capabilities.

read the caption

Figure 3: Illustration of various post-training datasets. Models can post-train on various vision-language datasets using a unified tokenizer and support diverse vision-language applications, such as question answering, image captioning, image/video question answering, visual grounding (including points and bounding box), and decision-making. More examples can be found in Appendix D.

🔼 This figure displays the results of ablation studies conducted to determine the impact of different post-training datasets on downstream decision-making tasks. Three types of datasets were used: knowledge datasets, visual question-answering datasets, and spatial grounding datasets. The results show the success rates for three downstream tasks (crafting a sword, mining obsidian, and cooking beef) when different combinations of post-training datasets were used. This illustrates which visual-language capabilities are most crucial for effective task performance, highlighting the importance of a well-rounded, multi-faceted post-training process.

read the caption

Figure 4: Ablation results on different post-training datasets. We select knowledge datasets, visual question-answering datasets, and spatial grounding datasets to conduct ablation experiments. Our goal is to evaluate which capabilities and post-training datasets most significantly influence downstream decision-making tasks.

🔼 Figure 5 illustrates the relationship between the success rate of downstream tasks, the training loss, and the number of training steps in a model. The graph shows that increasing the number of downstream fine-tuning trajectories improves the model’s success rate, but only when the training loss is below 0.22. This indicates a threshold where additional data significantly impacts performance.

read the caption

Figure 5: The relation between downstream task success rate, training loss, and training steps. The curve shows that scaling downstream finetuning trajectories can scale up the success rate when the loss is lower than 0.22.

🔼 This figure illustrates the correlation between the loss during post-training on non-trajectory datasets and the success rate on downstream tasks. It demonstrates that increasing the size of the non-trajectory training data improves the performance on downstream tasks, even when the amount of fine-tuning trajectory data remains constant. This highlights the importance of comprehensive pre-training in enhancing the model’s ability to generalize to new tasks.

read the caption

Figure 6: The relationship between post-training loss and downstream task success rates. Our findings indicate that increasing the size of post-training non-trajectory datasets can significantly enhance downstream task success rates, even with a fixed number of fine-tuning trajectories.

🔼 This table details the evaluation dataset for the World Knowledge portion of the Minecraft benchmark. It breaks down 47 questions into eight subcategories (Craft, Plant, Mobs, Kill, Enchant, Potions, Biomes, Architecture), showing the number of questions per category and example questions and answers. This provides a detailed look at the types of knowledge questions used to assess the models’ understanding of the Minecraft world.

read the caption

Table 2: Summary of Minecraft knowledge questions and answers Evaluation.

Category	Quantity	Image	Example Question	Example Answer
Scene	11		What biome you think is the player currently in?	Mushroom Fields biome.
Object	13		Are there any hostile mobs in the picture?	No, there are only 4 pigs in the picture.
Info	7		Is the player’s hunger bar currently full?	No.
Inventory	6		Is there any oak wood in the inventory?	Yes, there are oak wood planks in the inventory.
OCR	6		What instructions are visible on the screen?	Open your inventory Press e.

🔼 This table details the evaluation metrics for the Vision Understanding portion of the Minecraft benchmark. It breaks down the evaluation into five subcategories: Scene, Object, Info, Inventory, and OCR. Each category contains several questions related to aspects of the Minecraft game environment and the player’s status. The table lists the number of questions per category, provides example questions and answers, and includes an image to illustrate the visual information used in each question.

read the caption

Table 3: Summary of Vision Understanding Evaluation.

Category	Quantity	Image	Example Question	Example Answer
GUI	100		Point the wheat_seeds	[284,206]
Embodied	236		Point the oak_leaves.	[315,174]

🔼 Table 4 presents a summary of the evaluation results for spatial grounding tasks in the paper. It shows the number of GUI (Graphical User Interface) and Embodied data points used for evaluation, provides example questions, and displays the format of the expected answers. The results are crucial for assessing the model’s ability to accurately locate objects within Minecraft scenes, differentiating between GUI elements and objects within the 3D game environment.

read the caption

Table 4: Summary of spatial grounding evaluation results for visual grounding tasks.

🔼 This table compares the performance of various Vision-Language Models (VLMs) on a benchmark designed to evaluate their knowledge of Minecraft, visual understanding, and spatial reasoning abilities. The models compared include both commercially available large language models (like GPT-4 and its smaller variant) and open-source models (such as Llava-Next, Molmo-d-0924, Llama-3.2, and Qwen2-VL). The table shows each model’s accuracy and ranking across these three categories. The key finding is that the JARVIS-VLA model, the authors’ approach, significantly improves the core capabilities of the other models, though a performance gap still exists relative to state-of-the-art models.

read the caption

Table 5: We compared the performance of various VLMs using our benchmark, including commercial large models (GPT-4 and GPT-4-mini [35]), open-source models (Llava-Next [27], Molmo-d-0924 [16], Llama-3.2 [33], and Qwen2-VL [43]), as well as JARVIS-VLA. The results demonstrate that our method significantly enhances the core capabilities of these models, although there remains a gap when compared to state-of-the-art models.

Methods	Models	World Knowledge	Visual Alignment	Spatial Grounding
Raw	Llava-Next-8B	18.9	41.8	-	26.7	10.0
ActVLP	Llava-Next-8B	55.8	60.3	-	53.3	16.6
Raw	Qwen2-VL-7B	17.3	46.5	16.6	83.3	0.0
ActVLP	Qwen2-VL-7B	70.7	76.7	88.0	86.7	83.3

🔼 This table presents the ablation study results, demonstrating the robustness of ActVLP across different vision-language models. ActVLP, the proposed Visual Language Post-Training method, is applied to two distinct base models: Llava-Next-8B and Qwen2-VL-7B. The table compares their performance on world knowledge, visual alignment, and spatial grounding tasks before and after applying ActVLP. It showcases the performance improvements achieved by ActVLP and indicates its effectiveness regardless of the underlying base model.

read the caption

Table 6: Ablation experiments on base model and model structure. We adopt ActVLP on Llava-Next-8B [27] and Qwen2-VL-7B [43] to validate the robustness across different base vision language models.