Multimodal Inconsistency Reasoning (MMIR): A New Benchmark for Multimodal Reasoning Models

🔼 Figure 2 presents five example inconsistencies from the MMIR benchmark dataset, each representing a different category of multimodal inconsistency. These examples illustrate the diverse challenges that the benchmark presents for multimodal reasoning models. The categories are: Factual Contradiction (a conflict between text and image), Identity Misattribution (mismatched brand information across different elements), Contextual Mismatch (thematic mismatch between elements), Quantitative Discrepancy (numerical errors in a chart compared to text), and Temporal/Spatial Incoherence (time or location conflicts within the presented elements). Each example demonstrates the complexity of detecting inconsistencies across modalities and different layouts.

read the caption

Figure 2: There are five inconsistency categories in the MMIR benchmark, posing diverse challenges.

🔼 The figure illustrates the four-stage process of MMIR data curation. It begins with the collection and parsing of 521 artifacts, followed by the synthetic injection of inconsistencies using the 01-1217 language model, resulting in 2,534 proposals. After automated editing and human verification, the final dataset of 534 validated quintuples is obtained. These quintuples consist of the modified artifact, its elements, the type of inconsistency introduced, and the rationale behind it. This filtered dataset forms the MMIR benchmark.

read the caption

Figure 3: MMIR Data filtering process.

🔼 This figure presents a detailed breakdown of model performance on the MMIR benchmark. Panel (a) shows accuracy broken down by the five types of inconsistencies (factual contradiction, identity misattribution, contextual mismatch, quantitative discrepancy, and temporal/spatial incoherence). Panel (b) displays accuracy broken down by the type of modality involved in the inconsistency (text-only, image-only, text and image). The figure shows that models perform better on inconsistencies involving a single modality, particularly text, and struggle more with cross-modal inconsistencies. The superior performance of proprietary models (o1 and GPT-40) is also evident across all inconsistency categories and modalities.

read the caption

Figure 4: Fine-grained analysis of model performance.

🔼 This figure shows the relationship between the number of elements in an artifact and model performance on the MMIR benchmark. It reveals that as the complexity of the layout increases (i.e., more elements in the artifact), the accuracy of all models decreases. Proprietary models maintain higher accuracy in simpler layouts but show a similar decline in performance as complexity increases compared to open-source models. Open-source models struggle even with low-complexity artifacts.

read the caption

Figure 5: Model performance on layout complexity.

🔼 This figure shows a side-by-side comparison: on the left, the original webpage from the Multimodal Inconsistency Reasoning (MMIR) benchmark; on the right, the same webpage after it has been annotated using the Set-of-Marks (SoM) method for probing experiments. SoM highlights relevant elements (bounding boxes) in the image to help evaluate the effectiveness of prompting methods on visual multimodal reasoning. This helps assess how well models can leverage visual cues when integrated with textual prompting for improved reasoning performance.

read the caption

Figure 6: Example of original artifact in MMIR (left) and artifact annotated with Set-of-Mark in the probing analysis (right).

🔼 This figure displays a sample from the MMIR benchmark, illustrating how different models perform under open-ended and multiple-choice evaluation settings. The left side shows the original image, which contains a factual contradiction: a map labels a geographic region as ‘Rocky Mountain Range’ when it is in fact a different arctic region. The right side presents model responses for both question types; open-ended responses show the model’s ability to identify the error without predefined options, while multiple-choice responses test accuracy when provided with choices of potential inconsistent elements.

read the caption

Figure 7: A test sample with model responses under the two main settings in MMIR: open-ended and multiple-choice.

🔼 This table presents the performance of six different Multimodal Large Language Models (MLLMs) on a multimodal inconsistency reasoning benchmark (MMIR). The models are evaluated under two settings: an open-ended setting where the model freely generates a response identifying inconsistencies, and a multiple-choice question (MCQ) setting where the model selects from pre-defined options. The results show that proprietary models (those not open-source) significantly outperform open-source models in both settings, demonstrating a substantial performance gap. Moreover, proprietary models show a much larger performance improvement in the MCQ setting compared to the open-ended setting. This highlights the challenges posed by multimodal inconsistency reasoning and the limitations of current open-source MLLMs in this area.

read the caption

Table 2: The accuracy of six MLLMs under the two evaluation settings. Proprietary models demonstrate higher performance as well as larger performance gain in the MCQ setting.

🔼 This table presents the results of experiments evaluating different prompting methods on a multimodal inconsistency reasoning benchmark. The benchmark assesses the ability of large language models to detect inconsistencies in multimodal data. The methods tested include Chain-of-Thought (CoT) prompting, Set-of-Mark (SoM) visual augmentation, a combination of both, and a novel Multimodal Interleaved CoT (MM-CoT) approach. The table compares the performance of each method against a baseline (vanilla) setting, indicating gains or losses in accuracy. The results are categorized by model type (proprietary vs. open-source).

read the caption

Table 3: Probing results of different prompting methods. Performance of each prompting method is directly compared with the vanilla setting. Gains are in blue and drops are in red.