Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization

🔼 This figure showcases examples from the MMPR dataset, illustrating the two data generation pipelines used. The top half shows examples with clear ground truth. In these cases, the system generates multiple responses to the same prompt. Responses matching the ground truth are labeled as ‘chosen responses,’ while those that don’t match are ‘rejected responses.’ The bottom half presents examples where ground truth is unclear. Here, the ‘DropoutNTP’ method is used: the model generates a response, truncates it, and then attempts to complete the truncated response without the image context. This generated completion acts as the ‘rejected response’, while the original, complete response is the ‘chosen response’. Differences between chosen and rejected responses are highlighted in italics, and incorrect answers are highlighted in red.

read the caption

Figure 2: Data examples in MMPR. For instructions with clear ground truths, we propose a correctness-based pipeline, which samples multiple solutions and considers those with correct answers as chosen responses and those with incorrect answers as rejected responses. For instructions without clear ground truths, we propose DropoutNTP to generate rejected responses. Differences between the chosen and rejected responses are emphasized in italicized text. Red highlights incorrect responses.

🔼 This figure displays the performance comparison of various preference optimization algorithms applied to the M3CoT benchmark. The algorithms tested include Direct Preference Optimization (DPO), its variants (RSO, IPO, CDPO, RobustDPO, BCO, SPPO, AOT, TR-DPO), and a version of each extended by incorporating the Supervised Fine-Tuning (SFT) loss (indicated by ‘+’). The graph compares the accuracy results for each method when using direct answers and when using Chain-of-Thought (CoT) reasoning. This helps visualize how effectively each approach enhances the model’s ability to generate reasoned responses, highlighting the impact of the SFT loss on CoT reasoning performance.

read the caption

Figure 3: Results of models trained with different preference optimization algorithms on M3CoT. The algorithm X extended with the SFT loss is called X+ for brevity. For instance, DPO+ denotes the combination of DPO loss and SFT loss.

🔼 This figure shows examples from the MMPR dataset. Specifically, it presents examples of instructions with and without clear ground truth. For instructions with clear ground truths, a correctness-based pipeline is used which samples multiple solutions, labelling those matching the ground truth as ‘chosen responses’ and those that don’t as ‘rejected responses’. For instructions without clear ground truths, a DropoutNTP pipeline is used. The figure highlights the differences between chosen and rejected responses to illustrate how the dataset was constructed.

read the caption

(a)

🔼 This figure shows two magnets with their north and south poles labeled. The north poles of both magnets face each other, and the south poles of both magnets face each other. The caption indicates that this arrangement will cause the magnets to repel.

read the caption

(b)

🔼 The figure shows a bar chart visualizing the number of occurrences of different colors in the ‘fence’ row of a dataset. Each color represents a category (legs, index, engine, total), and the bar height indicates the value associated with that color in the ‘fence’ row. This illustrates the numerical data distribution within the specified row, revealing the relative frequencies or counts of each category.

read the caption

(c)

🔼 The figure shows examples from the MultiModal PReference (MMPR) dataset. Specifically, it displays examples from the OCR (Optical Character Recognition) category of the dataset. The top shows the question: ‘What is the total amount of this receipt?’, along with the chosen and rejected responses generated by a multimodal language model. The chosen response correctly calculates the total amount based on the invoice image, while the rejected response makes an incorrect calculation and includes extraneous information. This illustrates how the dataset differentiates between high-quality, correct reasoning and lower-quality, incorrect reasoning for multimodal tasks.

read the caption

(d)

🔼 This figure displays the performance of models trained on varying amounts of data and different hyperparameters, specifically focusing on the M3CoT benchmark. The graphs illustrate how changes in training data volume and hyperparameter settings (learning rate, PO coefficient, and SFT coefficient) affect the accuracy of the model. The x-axis of each graph shows the variable being tested (data scale or hyperparameter), while the y-axis indicates the accuracy achieved on the M3CoT benchmark. Separate lines represent the results for models making direct answers and those using chain-of-thought (CoT) reasoning.

read the caption

Figure 4: Results of models trained with different data scales or hyer-parameters on M3CoT. The X-axis represents the corresponding data scale or hyper-parameter for this point, while the Y-axis indicates the accuracy on M3CoT.

🔼 The figure shows examples of data samples from the MMPR dataset. Panel (a) shows an example with clear ground truth, using a correctness-based pipeline. Multiple response options to the same question are provided, where one response matches the ground truth and serves as the ‘chosen response’, while the others are incorrect and are labelled ‘rejected responses’. The differences between the chosen and rejected responses are highlighted in italicized text. Incorrect parts of rejected responses are highlighted in red. Panel (b) shows an example without clear ground truth, instead using a DropoutNTP pipeline. A generated response is truncated, and the model is prompted to complete it without image input. This incomplete response becomes the ‘rejected response’, while the original, complete response is the ‘chosen response’. Again, differences are highlighted in italicized text, and incorrect parts of the rejected response in red.

read the caption

(a)

🔼 This figure shows two bar magnets placed with their north poles facing each other. The caption explains the principles of magnetic attraction and repulsion: like poles repel, and unlike poles attract. The image is used to illustrate that because the north poles of both magnets are facing each other, the magnets will repel each other. This is a visual aid to support the explanation of magnetic forces in a section of the paper that discusses multimodal reasoning.

read the caption

(b)

🔼 The figure shows a bar chart visualizing the internet access percentage for five different countries in 2015. Each country is represented by a unique color, enabling easy differentiation between the countries and their internet access statistics. The chart’s structure makes it simple to compare the internet access levels across the selected nations.

read the caption

(c)

🔼 This figure shows examples from the MultiModal Preference dataset (MMPR). Specifically, it displays samples from the ‘Mathematics’ domain where the task is to determine the perimeter of triangle ABO given information about quadrilateral ABCD with intersecting diagonals. The ‘Chosen Response’ provides a step-by-step solution that leverages geometric properties and equations. The ‘Rejected Response’ demonstrates an alternative approach, highlighting the differences in reasoning processes and correctness in reaching the solution.

read the caption

(d)

🔼 This figure shows examples from the MMPR dataset. Specifically, it displays examples of instructions with and without clear ground truths. For instructions with clear ground truths, a correctness-based pipeline was used, where responses matching the ground truth are considered chosen responses and those that don’t match are rejected responses. For instructions without clear ground truths, the DropoutNTP pipeline was used. The chosen and rejected responses are presented, with differences highlighted to illustrate how the pipeline generates preference pairs for training. Red highlighting indicates incorrect responses.

read the caption

(e)

🔼 This figure shows examples from the MultiModal Preference dataset (MMPR). Specifically, it displays examples of instructions and their corresponding chosen and rejected responses from the OCR (Optical Character Recognition) domain. The chosen responses provide accurate interpretations of the image content, while the rejected responses contain errors or hallucinations. This visual aids in understanding the dataset’s composition and the differences between preferable and less preferable responses used for model training.

read the caption

(f)

🔼 This figure shows examples from the MultiModal Preference dataset (MMPR). Specifically, it displays examples of instructions and responses from the chart domain. The left side shows a ‘chosen response’ which is a correct and well-reasoned answer. The right side shows a ‘rejected response’ which is either incorrect or poorly reasoned. The responses are paired to illustrate the preference data used to train the model. The visual is a chart showing data points with different colors and the instruction asks about the color-coding of the chart and the value related to a specific color.

read the caption

(g)

🔼 This figure visualizes the results of models trained with different data scales on the M3CoT benchmark. The x-axis represents the size of the training dataset (10K, 40K, 70K, and 100K samples). The y-axis shows the accuracy achieved on the M3CoT benchmark. Two lines are plotted: one for the model’s performance using direct answers and the other for its performance using Chain-of-Thought (CoT) reasoning. The graph shows a clear positive correlation between the amount of training data and the model’s accuracy, indicating that scaling up the reasoning preference data improves the model’s performance.

read the caption

(h)

🔼 The figure displays the performance of various open-source multimodal large language models (MLLMs) on the MathVista benchmark. The x-axis represents accuracy with direct-answer responses, and the y-axis represents accuracy using Chain-of-Thought (CoT) reasoning. Each bubble represents a different model; the size of the bubble is correlated with the number of parameters in the model. The plot reveals a common trend among open-source MLLMs: performance decreases when CoT reasoning is used, indicating a distribution shift problem between training and inference. The authors’ model, InternVL2-8B-MPO, is shown to significantly outperform other models, achieving results comparable to much larger models.

read the caption

(i)

🔼 This table presents a comprehensive comparison of various multimodal large language models (MLLMs) across eight benchmark datasets. The benchmarks assess different aspects of MLLM capabilities, including reasoning, visual question answering (VQA), and hallucination. The table showcases the performance of several open-source MLLMs, including different variants of InternVL2, along with closed-source models like Gemini and GPT-4 for reference. The key finding is that InternVL2-8B-MPO, the model enhanced by the authors’ Mixed Preference Optimization (MPO) technique, significantly outperforms the base InternVL2-8B model, achieving performance on par with much larger models like InternVL2-76B.

read the caption

Table 2: Results on 8 multimodal benchmarks. We report the overall score of MM-Vet and LLaVA-Bench evaluated by GPT-4-Turbo. Our InternVL2-8B-MPO demonstrates superior performance compared to InternVL2-8B across multimodal reasoning, VQA, and hallucination evaluation benchmarks. Noteably, our model even achieves reasoning performance comparable to the 10×\times× larger InternVL2-76B.

🔼 This table compares the performance of two models: one trained with supervised fine-tuning (SFT) and another trained with the proposed mixed preference optimization (MPO) method. Both models are evaluated on their ability to answer questions directly (Direct) and using chain-of-thought (CoT) reasoning. The SFT model uses chosen responses from the MPO training data for its training. The table showcases the accuracy of both methods across multiple benchmark datasets, highlighting the difference in performance between Direct and CoT approaches for both training methods.

read the caption

Table 3: Results of models trained with SFT and MPO. The SFT training data consists of the chosen responses from the preference pairs used in MPO training. In the Direct setting, the model is prompted to provide the answer directly, while in the CoT setting, the model is instructed to answer with detailed rationales.

🔼 This table compares the performance of two methods for generating negative samples in a multimodal preference optimization dataset: DropoutNTP (a novel method proposed in this paper) and the divide-and-conquer approach from RLAIF-V (a previous work). Specifically, it shows the hallucination rates (in response-level and mention-level) on the Object HalBench benchmark, as well as the overall score and hallucination rates on the MMHalBench benchmark. The goal is to demonstrate the effectiveness of DropoutNTP, which achieves comparable performance to RLAIF-V’s more complex method, while being simpler and more efficient.

read the caption

Table 4: Comparison of DropoutNTP and the divide-and-conquer approach from RLAIF-V. We replace negative samples in RLAIF-V with the responses generated using DropoutNTP.

🔼 This table presents the results of evaluating the performance of language models on various text-only benchmarks. The models evaluated include a baseline model, a model fine-tuned with supervised fine-tuning (SFT), and a model fine-tuned using the proposed mixed preference optimization (MPO) method. The benchmarks cover a range of tasks, and the table shows the performance scores for each model on each benchmark. The MPO model demonstrates significantly improved performance compared to both the baseline model and the SFT model, especially on TheoremQA and IFEval.

read the caption

Table 5: Results on text-only benchmarks. The model fine-tuned through MPO exhibits superior overall performance on text-only tasks compared to the baseline model and its SFT counterpart, particularly on TheoremQA and IFEval.

🔼 This table presents the results of experiments evaluating various preference optimization algorithms applied to the M3CoT benchmark. For each algorithm (DPO, RSO, IPO, CDPO, RobustDPO, BCO, SPPO, AOT, TR-DPO), the table shows the accuracy scores achieved when the model generates responses directly (Direct) and when it uses chain-of-thought (CoT) reasoning. The final column, Δ, indicates the performance difference between CoT and direct responses, providing insights into how each optimization method affects the model’s reasoning capabilities. A positive Δ value suggests CoT improves the model while a negative Δ indicates CoT reasoning reduces performance.

read the caption

Table 6: Results of models trained with different preference optimization algorithms on M3CoT. ΔΔ\Deltaroman_Δ represents the performance gap between CoT responses and direct-answer responses.

🔼 This table presents the results of experiments on the M3CoT benchmark, comparing various preference optimization algorithms enhanced with supervised fine-tuning (SFT) loss. Each algorithm (DPO, RSO, IPO, CDPO, RobustDPO, BCO, SPPO, AOT, TR-DPO, ORPO) was tested in its original form and then with the addition of SFT loss (denoted by ‘+’). The table shows the performance of each model configuration in terms of accuracy on direct-answer and chain-of-thought (CoT) reasoning tasks, and calculates the difference (Δ) between the two. The purpose is to evaluate the effectiveness of incorporating SFT loss into different preference optimization approaches for improving the reasoning abilities of multimodal large language models, particularly in CoT scenarios.

read the caption

Table 7: Results of models trained with different preference optimization algorithms extended with SFT Loss on M3CoT. The algorithm X extended with the SFT Loss is called X+ for brevity. For instance, DPO+ is the combination of DPO and SFT loss.

🔼 This table presents a comparison of the performance of three different multimodal large language models (MLLMs) on eight benchmark datasets. The models are InternVL2-8B (baseline), InternVL2-8B-DPO+, InternVL2-8B-BCO+, and InternVL2-8B-MPO. InternVL2-8B-DPO+ and InternVL2-8B-BCO+ are fine-tuned using the Direct Preference Optimization (DPO) and Binary Classifier Optimization (BCO) methods, respectively. InternVL2-8B-MPO uses the Mixed Preference Optimization (MPO) method introduced in the paper. The benchmarks cover various multimodal reasoning tasks and hallucination evaluation. The results show that the MPO model significantly outperforms the baseline and the DPO+ and BCO+ models on most of the benchmarks.

read the caption

Table 8: Results of models trained with DPO+, BCO+ and MPO using our MMPR.

🔼 This table presents an ablation study on the effect of varying the dropout ratio in the Dropout Next-Token Prediction (DropoutNTP) method used for generating negative samples in the MMPR dataset. The dropout ratio determines the portion of a positive response that’s truncated before the model attempts completion. The table shows the impact of using different dropout ratios (0.25, 0.50, and 0.75) on the hallucination rates measured by two different metrics: response-level and mention-level hallucinations. The results are presented for the Object HalBench and MMHalBench datasets.

read the caption

Table 9: Results of DropNTP with different Dropout Ratios.