LLaVA-o1: Let Vision Language Models Reason Step-by-Step

🔼 This figure showcases a comparison between the reasoning capabilities of two models: Llama-3.2-11B-Vision-Instruct (the base model) and LLaVA-01. Two example problems are presented, each involving visual reasoning. The base model demonstrates significant flaws and errors in its reasoning process, often producing inaccurate or illogical steps. In contrast, LLaVA-01 exhibits a systematic and structured approach. It starts by summarizing the problem, then extracts relevant information from the image, meticulously outlines a step-by-step reasoning process, and finally arrives at a logically sound and well-supported conclusion. This highlights LLaVA-01’s superior ability to perform systematic and structured reasoning compared to the base model.

read the caption

Figure 2: Comparison of the base model and LLaVA-o1. As shown, the base model Llama-3.2-11B-Vision-Instruct exhibits obvious flaws in reasoning, with several errors occurring throughout the reasoning process. In contrast, LLaVA-o1 begins by outlining the problem, interprets relevant information from the image, proceeds with a step-by-step reasoning process, and ultimately reaches a well-supported conclusion.

🔼 This figure illustrates the process of creating the LLaVA-01-100k dataset. The process starts with a question and involves four stages: 1. Summary: GPT-40 summarizes the question and outlines the overall approach. 2. Caption: If an image is part of the question, GPT-40 describes the relevant visual elements. 3. Reasoning: GPT-40 outlines a step-by-step logical reasoning process to answer the question. 4. Conclusion: GPT-40 provides the final answer. The outputs from each stage are then filtered to ensure high quality before being included in the dataset.

read the caption

Figure 3: Process flow for generating the LLaVA-o1-100k dataset. We prompt GPT-4o to generate responses in separate stages, and filter its outputs to ensure quality.

🔼 Figure 4 illustrates three different inference time scaling methods: Best-of-N search, sentence-level beam search, and the proposed stage-level beam search. Best-of-N search generates multiple complete answers and selects the single best one. This approach is computationally expensive and may not be effective when responses vary widely in quality. Sentence-level beam search generates multiple options for each sentence and chooses the best among them. This approach is quite granular, focusing on small portions of the text and potentially missing important contextual relationships. In contrast, the paper’s proposed stage-level beam search generates candidates for each stage of the reasoning process (summary, caption, reasoning, and conclusion) and selects the best option at each stage. By focusing on the broader reasoning structure and checking the quality of each step, it offers a better balance between efficiency and accuracy. The figure highlights that the stage-level approach achieves superior performance compared to the other two methods due to its optimal granularity.

read the caption

Figure 4: An illustration of inference approaches. Best-of-N search generates N𝑁Nitalic_N complete responses and selects the best one among them; Sentence-level Beam Search generates multiple candidate options for each sentence and chooses the best one. In contrast, our Stage-level Beam Search generates candidates for each reasoning stage (e.g., summary, caption, reasoning, and conclusion) and selects the best option at each stage. Best-of-N search operates at a coarse level, while Sentence-level Beam Search is overly granular, and our method achieves an optimal balance and achieves the best performance.

🔼 The figure showcases a comparison of LLaVA-01’s performance on a visual question answering task, both with and without the application of a stage-level beam search. Two examples of question-answering tasks are presented: one involving a simple counting problem and another involving a physics problem that necessitates a step-by-step reasoning process. For each problem, the figure displays the base model’s answer (Llama-3.2-11B-Vision-Instruct) and LLaVA-01’s answer. LLaVA-01’s answer shows the model’s step-by-step reasoning process through four distinct stages: summarization, captioning, reasoning, and conclusion. The base model’s answer is presented as a single step without explicit reasoning, showing its limitations in handling complex reasoning tasks. In contrast, LLaVA-01 demonstrates more robust reasoning by outlining the problem, interpreting relevant information from the image, engaging in structured step-by-step reasoning, and finally, providing well-supported conclusions. The comparison highlights that the stage-level beam search in LLaVA-01 is crucial for effective inference, enabling more accurate and systematic solutions to complex problems.

read the caption

Figure 5: Comparison of LLaVA-o1 performance with and without stage-level beam search. Our stage-level beam search is effective in selecting better reasoning during model inference.

🔼 This table presents a comparison of the performance of different models on a multimodal reasoning benchmark. Three variations of the LLaVA-01 model are included: one trained directly on the original VQA dataset (without the structured reasoning stages), one trained on the LLaVA-01-100k dataset but without the structured tags used to denote reasoning stages, and a final version trained on the complete LLaVA-01-100k dataset with the structured tags. A baseline model (Llama-3.2-11B-Vision-Instruct) is also included for comparison. The results highlight the impact of the structured training data and tags on the model’s performance.

read the caption

Table 2: Experimental results of different models on the benchmark. Here, LLaVA-o1 (with Direct Training) refers to the model trained directly on the original VQA dataset’s Q&A pairs, while LLaVA-o1 (w/o Structured Tags) represents the model trained on the LLaVA-o1-100k dataset with the structured tags removed. LLaVA-o1 refers to the model trained on the complete LLaVA-o1-100k dataset, including the structured tags.

🔼 Table 3 presents a detailed comparison of different models’ performance on the MMStar benchmark, broken down by specific skill areas: Coarse Perception (CP), Fine-grained Perception (FP), Instance Reasoning (IR), Logical Reasoning (LR), Math, and Science & Technology. The results highlight LLAVA-01’s significant improvement over the baseline model, particularly in the more complex reasoning tasks (IR, LR, Math, and Science & Technology), demonstrating the effectiveness of its structured reasoning approach in enhancing overall reasoning capabilities.

read the caption

Table 3: Performance of different models on the MMStar benchmark across various skill areas. Here, CP represents coarse perception, FP represents fine-grained perception, IR represents instance reasoning, and LR represents logical reasoning. As shown in the table, our model demonstrates substantial improvement over the base model in instance reasoning, logical reasoning, math, and science & technology, indicating that structured reasoning can significantly enhance the model’s reasoning capabilities.

🔼 This table presents the performance comparison of different models during inference time. Specifically, it contrasts the performance of the LLaVA-01 model without any inference-time scaling techniques, against the same model using a stage-level beam search with a beam size of 2 (LLaVA-01 (BS=2)). The results highlight the significant performance gains achieved by employing the stage-level beam search method, demonstrating its effectiveness in improving the model’s reasoning capabilities during inference.

read the caption

Table 4: Experimental results during inference time. LLaVA-o1 (BS = 2) denotes the model using stage-level beam search with a beam size of 2. The results show that stage-level beam search can achieve further significant performance improvements.

🔼 Table 7 presents a comparative analysis of the performance of LLaVA-01 and other state-of-the-art vision-language models (VLMs) across six reasoning benchmarks. These benchmarks assess various reasoning capabilities, including general visual question answering, mathematical reasoning, scientific reasoning, and handling of hallucinations and visual illusions. The table specifically contrasts the performance of LLaVA-01 without inference-time scaling and LLaVA-01 with a stage-level beam search (using a beam size of 2). This comparison highlights the impact of the proposed inference-time scaling technique on the overall performance of the model.

read the caption

Table 7: Experimental results of LLaVA-o1 and state-of-the-art models on reasoning benchmarks. Here, LLaVA-o1 refers to the model without inference scaling, while LLaVA-o1 (BS = 2) denotes the model using stage-level beam search with a beam size of 2.