URSA: Understanding and Verifying Chain-of-thought Reasoning in Multimodal Mathematics

🔼 This figure shows the data sources used to train the URSA-7B model. It breaks down the composition of the URSA-alignment-960K dataset used in the vision-language alignment phase and the MMathCoT-1M dataset used in the subsequent supervised fine-tuning (SFT) phase. The figure visually represents the percentage contribution of each dataset to the overall training data. It highlights the diversity of sources, including datasets focusing on various mathematical problem types and formats (e.g., geometry, word problems, and tables).

read the caption

Figure 2: Data sources used by the URSA-7B model during the VL-alignment and SFT phases.

🔼 This figure illustrates the process of generating high-quality chain-of-thought (CoT) reasoning data for multimodal mathematics. It uses Gemini-1.5-Flash-002 to augment existing datasets in three ways: CoT distillation, trajectory rewriting, and format unification. The figure shows examples of how input data from three different sources is transformed using these techniques to create a more consistent and comprehensive CoT dataset for training and validating multimodal mathematical reasoning models. Each step demonstrates how Gemini-1.5-Flash-002 is used to enhance the reasoning process, leading to a unified, improved CoT format suitable for fine-tuning.

read the caption

Figure 3: CoT augmentation and verifying for multimodal mathematical data from three type of sources using Gemini-1.5-Flash-002.

🔼 This figure demonstrates the process of inserting misinterpretations into a reasoning chain. The example shows a chart about public opinion on labor unions, broken down by age, education level, and political leaning. The model initially extracts information correctly, identifying that certain groups have more favorable views toward unions. However, a misinterpretation is then introduced, such as incorrectly interpreting the data for one group. Following the misinterpretation, the model continues to reason and draw incorrect conclusions based on this flawed interpretation. This illustrates how the engine creates training data by highlighting logical errors stemming from visual misinterpretations.

read the caption

Figure 4: Demonstration of Misinterpretation Insertion Engine.

🔼 This figure illustrates the three-stage training process for the URSA-7B and URSA-RM-7B models. Stage 1 involves vision-language alignment using the URSA-alignment-960K dataset, focusing on aligning the vision encoder and language model. Stage 2 performs mathematical instruction fine-tuning on the MMathCoT-1M dataset, enhancing the model’s chain-of-thought (CoT) reasoning capabilities. Finally, Stage 3 employs dual-view process supervision using the DualMath-1.1M dataset, training a verifier model (URSA-RM-7B) to enhance the reasoning at test time. The figure clearly shows which model components are frozen and which are trained during each stage.

read the caption

Figure 5: An illustration of the training process for URSA-7B and URSA-RM-7B, with data from the three stages coming from URSA-alignment-960K, MMathCoT-1M, and DualMath-1.1M, respectively. The modules that are frozen and those that need to be trained are distinguished in each stage.

🔼 This figure compares the performance of different methods for improving the accuracy of chain-of-thought reasoning in mathematical problem-solving, specifically focusing on two benchmarks: MathVerse and MathVista-GPS. It shows how the ‘pass@N’ metric (the percentage of times a model correctly answers a question within N attempts) and the ‘best-of-N’ metric (choosing the best answer among N attempts) change with the number of samples (N). The methods compared likely include the baseline model, a self-consistency approach, and the URSA-RM-7B verifier model, which are described in the paper. The results visualize how using multiple samples and the verifier can lead to significant improvements in accuracy compared to relying on a single answer.

read the caption

Figure 6: Pass@N and Best-of-N results comparison on MathVerse and MathVista-GPS.

🔼 This figure presents the results of an ablation study investigating the impact of chain-of-thought (CoT) augmentation during the mathematical instruction fine-tuning (Math SFT) stage on the MathVista testmini dataset. It shows how the performance of the model changes when CoT augmentation is removed, providing a quantitative analysis of the contribution of CoT reasoning to the model’s capabilities on various subtasks within MathVista. This allows for a better understanding of the effectiveness of the CoT augmentation strategy.

read the caption

Figure 7: Ablation Study w.r.t CoT Augmentation during Math SFT Stage on the MathVista testmini Set.

🔼 This figure shows the ablation study with respect to chain-of-thought (CoT) augmentation during the mathematical instruction fine-tuning (Math SFT) stage on the MathVerse benchmark. It visually compares the performance of the model with and without CoT augmentation across various sub-tasks within the MathVerse benchmark. The graph likely shows accuracy or a similar performance metric on the y-axis, and different sub-tasks or aspects of the MathVerse benchmark on the x-axis. This allows readers to see how much the addition of CoT augmentation improves the model’s performance on each MathVerse sub-task.

read the caption

Figure 8: Ablation Study w.r.t CoT Augmentation during Math SFT Stage on the MathVerse benchmark.

🔼 Prompt 𝒫𝒞 for CoT distillation is used to generate chain-of-thought (CoT) reasoning from answer-only data. The prompt instructs the model to produce a step-by-step solution that leads to the given answer, emphasizing the importance of maintaining consistency between the generated reasoning and the provided answer. The prompt also ensures that the model doesn’t request additional information or express uncertainty, focusing instead on a clear and concise solution.

read the caption

Figure 9: Prompt 𝒫𝒞subscript𝒫𝒞\mathcal{P}_{\mathcal{C}}caligraphic_P start_POSTSUBSCRIPT caligraphic_C end_POSTSUBSCRIPT used for CoT distillation on answer-only source data.

🔼 This figure shows the prompt used in the paper for the CoT solution rewriting task on analysis-formatted data. The prompt instructs the model to rewrite a given solution while maintaining the correctness of both the process and the final answer. It emphasizes the need for semantically coherent transcription and forbids the model from making requests or altering parts of the given solution.

read the caption

Figure 10: Prompt 𝒫ℛsubscript𝒫ℛ\mathcal{P}_{\mathcal{R}}caligraphic_P start_POSTSUBSCRIPT caligraphic_R end_POSTSUBSCRIPT used for CoT solution rewriting on analysis-formatted source data.

🔼 Prompt 𝒫ℱ used for unifying solution format across style-varied source data. This prompt is used in the data synthesis process for the instruction fine-tuning of the URSA-7B model. The prompt instructs the model to convert a formal mathematical solution into a natural language explanation, ensuring clarity, conciseness, and adherence to the original solution’s steps and final answer, avoiding modifications or reinterpretations.

read the caption

Figure 11: Prompt 𝒫ℱsubscript𝒫ℱ\mathcal{P}_{\mathcal{F}}caligraphic_P start_POSTSUBSCRIPT caligraphic_F end_POSTSUBSCRIPT used for unifying solution format across style-varied source data.

🔼 This figure shows the prompt used for evaluating the quality of chain-of-thought (CoT) augmentation responses. The prompt guides the evaluator to assess the response based on two key criteria: solution fidelity (whether the reasoning is sound and free of speculation, and whether the final conclusion matches the given standard answer) and solution consistency (whether the reasoning steps logically lead to the answer without discrepancies). The evaluator is instructed to provide a judgment of ‘yes’ or ’no’ based on this evaluation.

read the caption

Figure 12: Prompt used for checking CoT augmentation response based on consistency and correction.

🔼 This figure presents the prompt used for the misinterpretation insertion engine in the Dual-view Process Supervised Data Synthesis stage. The prompt instructs the model to introduce errors into a geometry problem solution by misreading the diagram and producing an incorrect answer. The process involves three stages: analyzing the correct solution, introducing a misinterpretation, and continuing reasoning with the misinterpretation to arrive at an incorrect answer. The prompt emphasizes that the misreading should be integrated naturally into the solution without explicit statements about making a misinterpretation, and the final answer should be marked with ‘†Answer:’.

read the caption

Figure 13: Prompt used for inserting interpretation into geometry-related samples.

🔼 This figure shows the prompt used in the misinterpretation insertion engine for function and statistics related samples. The prompt instructs the model to introduce errors into a solution by misinterpreting a coordinate axis or chart. The model must identify areas in the solution where diagram information is extracted, choose one area to introduce a misinterpretation, and continue the reasoning process to derive an incorrect answer. The response should be natural and avoid explicitly mentioning misinterpretations. The prompt includes tags to mark correct and incorrect steps and specifies that the final answer should not be tagged. The misinterpretation action must be consistent with the plan outlined in the prompt.

read the caption

Figure 14: Prompt used for inserting interpretation into function and statistics-related samples.

🔼 This figure shows a geometry problem from the MathVista-GPS dataset. The problem involves a quadrilateral ABCD where AD = 6, AB = 4, and DE bisects angle ADC, intersecting BC at point E. The question asks for the length of BE. The figure displays several different attempts to solve this problem using different methods. These attempts include a correct solution, and solutions produced by GPT-40, Gemini-1.5-Flash-002, MultiMath-7B, Math-LLaVA-13B, and URSA-7B. The figure highlights the different approaches and results obtained by each method, illustrating the variety of reasoning paths used to solve geometry problems and the different levels of accuracy achieved.

read the caption

Figure 15: Case on MathVista-GPS.

🔼 The figure displays a geometry problem from the MathVerse benchmark dataset. The problem involves a circle with a tangent line and several angles. The question asks for the length of a specific line segment (AP), given the length of another line segment (OP) and the measure of a particular angle (∠BOC). Different models’ solutions and their correctness are compared and illustrated.

read the caption

Figure 16: Case on MathVerse.

🔼 Table 2 presents a comprehensive comparison of various Large Language Models (LLMs) on the WE-MATH testmini dataset, focusing on four key performance metrics: Average (AVG), Insufficient Knowledge (IK), Inadequate Generalization (IG), and Complete Mastery (CM). The models are categorized into closed-source and open-source LLMs, with the top-performing closed-source models highlighted in green and the top two open-source models highlighted in red and blue, respectively. This allows for a nuanced understanding of each model’s strengths and weaknesses in different aspects of mathematical reasoning, providing valuable insights into the current state-of-the-art in multimodal mathematical reasoning.

read the caption

Table 2: The performance comparison with Closed-source MLLMs and Open-source MLLMs on four-dimensional metrics for WE-MATH testmini reasoning evaluation. The best results of Closed-source MLLMs are highlighted in green. The best and second-best results of Open-source MLLMs are highlighted in red and blue respectively.

🔼 This table compares the performance of URSA-7B against other open-source large language models (LLMs) specifically designed for mathematical reasoning on the DYNAMATH testmini dataset. DYNAMATH is a benchmark designed to evaluate the robustness of LLMs in multimodal mathematical reasoning across different mathematical skill areas, problem types and difficulty levels. The table presents performance scores across various subtests of the DYNAMATH dataset, allowing for a detailed comparison of the models’ abilities in different mathematical reasoning tasks.

read the caption

Table 3: Comparison with open-source MLLMs on DYNAMATH testmini dataset.

🔼 This table presents the out-of-distribution (OOD) performance of the URSA-RM-7B model, which acts as a verifier, when evaluated on the Multimath-7B dataset. It compares the accuracy of URSA-RM-7B against a baseline method (Self-Consistency) for different sampling numbers (N). The results show how effectively URSA-RM-7B can enhance the reasoning capabilities of URSA-7B by identifying correct reasoning trajectories, particularly in OOD scenarios.

read the caption

Table 4: OOD performance when URSA-RM-7B works on Multimath-7B.

🔼 This table presents the results of an ablation study conducted on the URSA-RM-7B model. The study investigates the impact of removing specific components of the model’s training process on its performance. Specifically, it compares the performance of the full URSA-RM-7B model against versions where either the BinaryErrorLocating (SBEL) engine or the Misinterpretation Insertion Engine (SMIE) are removed. Results are reported for two separate benchmarks, MathVista-GPS and MathVerse, indicating how the removal of each component affects overall accuracy.

read the caption

Table 5: Ablation study on URSA-RM-7B.

🔼 This table details the hyperparameters used and the training time taken for each of the three stages in the training process of the URSA-7B model. It includes information such as learning rate, number of epochs, batch size, and which parts of the model were trained at each stage. The time cost is also provided for each stage.

read the caption

Table 6: Hyperparameter setting and training time cost.

🔼 Table 7 presents a comparison of the performance of various Large Language Models (LLMs) on the MATHVISTA testmini dataset, focusing on their mathematical reasoning abilities. It compares both closed-source and open-source LLMs, providing a detailed breakdown of their accuracy across different mathematical subtasks (Algebra, Arithmetic, Geometry, Logic, Number, Science, Statistics). The table helps in assessing the relative strengths and weaknesses of different models in various mathematical domains.

read the caption

Table 7: Comparison with close-source MLLMs open-source MLLMs on MATHVISTA testmini mathematics capabilities.

🔼 Table 8 presents a detailed comparison of the performance of various Large Language Models (LLMs) on the WE-MATH testmini subset. The table is organized to show the overall accuracy of each model across different problem complexities (one-step, two-step, and three-step problems). Additionally, it provides a more granular breakdown of performance based on different problem-solving strategies, offering insights into the strengths and weaknesses of each model in various mathematical reasoning approaches.

read the caption

Table 8: Accuracy comparison with close-source MLLMs and open-source MLLMs on WE-MATH testmini subset. First 3 columns show the overall performance on one-step, two-step and three-step problems. The other columns are used to demonstrate the performance in different problem strategies.

🔼 This table compares the performance of various large language models (LLMs) on the DYNAMATH testmini dataset, categorized by knowledge level (elementary school, high school, undergraduate). It shows the overall accuracy of each model at these three knowledge levels, allowing for comparison of model performance across different levels of mathematical complexity.

read the caption

Table 9: Comparison with close-source MLLMs open-source MLLMs on DYNAMATH testmini based on knowledge level.

🔼 Table 10 presents a performance comparison of various Multimodal Large Language Models (MLLMs) on the GeoQA benchmark. GeoQA specifically tests the ability of MLLMs to solve geometric reasoning problems. The table compares the accuracy of different models, including both closed-source and open-source models, across varying model sizes (parameters). This allows for an assessment of the impact of model size on geometric reasoning capabilities.

read the caption

Table 10: Performance comparison of different MLLMs on GeoQA.