DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models

🔼 The figure illustrates the process of generating a dynamic benchmark dataset for evaluating the robustness of vision-language models (VLMs) in mathematical reasoning. It starts with a seed question, represented as a Python program. This program generates numerous concrete question variants by randomly altering parameters (numerical values, function types, etc.), producing different visual representations (plots, graphs, etc.). Each variant has a corresponding ground-truth answer. During the evaluation phase, all generated variants of each seed question are used to assess the model’s performance, enabling the calculation of both average-case and worst-case accuracy, providing a comprehensive measure of robustness against variations.

read the caption

Figure 2: The dynamic benchmark generation procedure in DynaMath. A seed question is represented as a program that can generate many concrete questions with different variations. The plots for concrete questions are randomly generated along with the corresponding ground-truth answers. During evaluation, all concrete variants of the seed questions are considered, allowing us to evaluate the worst-case model performance and robustness.

🔼 This figure compares the reasoning robustness of various vision-language models (VLMs) across different aspects. The top panel shows the overall reasoning robustness of each model, indicating how consistently each model performs across various question variants. The middle panel breaks down the robustness performance across different math problem topics, showing variations in the models’ abilities across diverse mathematical domains. The bottom panel analyzes the robustness concerning various types of question variations, assessing how sensitive the models are to changes in numerical values, geometric transformations, functional representations, and so on.

read the caption

Figure 5: Comparing reasoning robustness across different models (top), topics (middle), and variant types (bottom).

🔼 Figure 6 demonstrates the memorization phenomenon observed in Claude 3.5 Sonnet. Five variants of seed question 12, each with a different visual representation of a periodic function, were generated. Despite the varying inputs, the model consistently predicted the period of the function as 2π. This indicates that instead of performing actual calculations based on the diagram’s details, the model may be relying on memorized patterns or heuristics. The high probability of the model giving the same answer, regardless of visual changes in the input, highlights a significant limitation in its reasoning capability and emphasizes the need for more robust evaluation of vision-language models.

read the caption

Figure 6: Example of the Memorization Phenomenon: the generated variants of seed Question 12 and the corresponding responses from Claude 3.5 Sonnet. The model’s response remains 2⁢π2𝜋2\pi2 italic_π with high probability, regardless of changes in the conditions depicted in the diagram.

🔼 The figure shows a pie chart that breaks down the types of errors made by the Claude-3.5 Sonnet model on the DYNAMATH benchmark. It visually represents the proportion of errors attributed to five categories: figure reading errors, calculation errors, reasoning errors, knowledge errors, and hallucination errors. This allows for a quick understanding of the model’s failure modes and their relative frequencies.

read the caption

Figure 7: Error Analysis of Claude-3.5 Sonnet.

🔼 Figure 7 visualizes six distinct variation types incorporated within the DynaMath benchmark. These variations manipulate different aspects of mathematical problems to assess the robustness of Vision-Language Models (VLMs). The variations include altering numerical values, performing geometric transformations, modifying function types, applying symbolic substitutions, incorporating real-life contexts, and changing graph structures. Each variation type challenges VLMs’ ability to generalize their reasoning processes across diverse problem instances.

read the caption

Figure 8: Variation types considered in our DynaMath benchmark

🔼 Figure 9 shows six variations of Question 169 from the DynaMath benchmark. Question 169 asks whether the product of two functions, f(x) and g(x), represented graphically, is even or odd. Each variant displays a slightly altered version of the graphs of f(x) and g(x), testing the model’s robustness to changes in visual representation. The figure also includes the corresponding answers generated by GPT-40 for each variant. The differences in the answers highlight GPT-40’s inconsistency in solving similar problems with minor visual changes.

read the caption

Figure 9: Example of the generated variants of Question 169 and the corresponding responses from GPT-4o.

🔼 Figure 10 presents six variations of Question 75 from the DYNAMATH benchmark, each showing different visual representations of two lines. The question asks whether the lines are parallel. Gemini’s responses to each variant are included, demonstrating inconsistencies in its ability to correctly assess parallelism based on these different visual presentations.

read the caption

Figure 10: Example of the generated variants of Question 75 and the corresponding responses from Gemini.

🔼 Table 2 presents the average-case accuracy of various vision-language models (VLMs) on the DynaMath benchmark. DynaMath consists of 5,010 dynamically generated visual math questions, derived from 501 seed questions. The table shows the performance of each model across different question topics (Plane Geometry (PG), Solid Geometry (SG), Analytic Geometry (AG), Algebra (AL), Puzzle Tests (PT), Graph Theory (GT), Statistics (ST), Scientific Figures (SF), Arithmetic (AR)), and difficulty levels (Elementary school (EL), High school (HI), Undergraduate (UN)). The ‘ALL’ column shows the overall average accuracy across all questions. The results are useful for comparing the performance of different models on various types of visual mathematical reasoning tasks and assessing their strengths and weaknesses.

read the caption

Table 2: Average-case accuracy 𝒜a⁢v⁢gsubscript𝒜𝑎𝑣𝑔\mathcal{A}_{avg}caligraphic_A start_POSTSUBSCRIPT italic_a italic_v italic_g end_POSTSUBSCRIPT on DynaMath with 5,010 generated questions. “ALL” represents overall accuracy. Question topics and difficulty levels (PG, EL, etc) are defined in Table 1.

🔼 Table 3 presents the worst-case accuracy (the lowest accuracy across 10 variations of each question) of various vision-language models (VLMs) on the DynaMath benchmark. It shows the performance of each model on different mathematical question types and difficulty levels (Elementary, High School, Undergraduate) as well as an overall worst-case accuracy. The table helps assess how robust each model is to variations in question presentation, emphasizing its ability to generalize. The question types and difficulty levels are defined in Table 1 of the paper.

read the caption

Table 3: Worst-case accuracy 𝒜w⁢s⁢tsubscript𝒜𝑤𝑠𝑡\mathcal{A}_{wst}caligraphic_A start_POSTSUBSCRIPT italic_w italic_s italic_t end_POSTSUBSCRIPT on DynaMath with 5,010 generated questions. “ALL” represents overall accuracy. Question topics and difficulty levels (PG, EL, etc) are defined in Table 1.

🔼 This table presents the repetition consistency (RC) scores for various vision-language models. Repetition consistency measures the consistency of a model’s responses to the same question across multiple repetitions. A higher RC indicates greater confidence and less inherent randomness in the model’s answers. The results are calculated from 5 repetitions for each question in the dataset. The table helps assess the reliability of each model, identifying those that provide consistent answers even when facing the same prompt multiple times.

read the caption

Table 4: The Repetition Consistency (R⁢C𝑅𝐶RCitalic_R italic_C) for different models over 5 repetitions.

🔼 This table presents the different prompts used for generating answers based on the question type. The prompt engineering approach is tailored to guide the model to produce responses in specific formats, depending on whether the question is multiple-choice, requires a numerical (floating-point) answer, or needs a text-based response. This ensures consistency and facilitates accurate evaluation of the model’s performance.

read the caption

Table 5: The prompt for different questions and answer types in answer generation.

🔼 This table lists the hyperparameters used for different Vision-Language Models (VLMs) during the experiments in the paper. For each model, it specifies the model name, the specific model version used (e.g., model size), the temperature setting, which controls the randomness of the model’s outputs, and the maximum number of tokens allowed in the model’s response.

read the caption

Table 6: Hyperparameters for various VLMs.