Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey

🔼 This figure presents a comprehensive taxonomy of Multimodal Chain-of-Thought (MCoT) reasoning methods. It organizes various MCoT techniques across several dimensions, including the modality of data involved (image, video, audio, 3D, table/chart, cross-modal), the methodology employed (rationale construction, structural reasoning, information enhancing, objective granularity, multimodal rationale, test-time scaling), and the applications in which these methods are used (embodied AI, agentic systems, autonomous driving, healthcare, social and human interaction, and multimodal generation). Each category contains specific examples of relevant models and papers.

read the caption

Figure 2: Taxonomy of MCoT reasoning.

🔼 This figure illustrates the evolution of reasoning structures or topologies used in Chain-of-Thought (CoT) and Multimodal Chain-of-Thought (MCoT) reasoning. It starts by showing basic paradigms for CoT and MCoT, highlighting the differences between direct input-output methods and methods that employ majority voting or self-consistency checks. Then, it moves to structured CoT and MCoT methods, which employ more sophisticated reasoning structures. Specifically, the figure displays three main types of topologies: chain, tree, and graph. Chain topologies represent linear, sequential reasoning; tree topologies allow for exploration and backtracking; and graph and hypergraph topologies facilitate aggregation among multiple nodes, enabling reasoning over more complex multimodal inputs. The figure showcases how the evolution of these topologies directly reflects a progression from linear reasoning to parallel and branching exploration with higher-order associations.

read the caption

Figure 3: Different thought paradigms of CoT and MCoT.

🔼 Figure 4 illustrates the typical architectures of multimodal large language models (MLLMs). It contrasts comprehension-only MLLMs with those capable of both comprehension and generation. Comprehension-only models process multimodal embeddings or tokens into a decoder structure that produces an output. Comprehension and generation models utilize a similar process but also include a generation component allowing them to produce both multimodal outputs (text, image, video, audio, 3D data) and textual outputs.

read the caption

Figure 4: Common architectures for comprehension-only and comprehension-generation MLLMs.

🔼 Figure 5 presents several example applications of Multimodal Chain-of-Thought (MCoT) reasoning across various modalities. Each example showcases a task, the input modalities (e.g., audio, image, video), the reasoning process as a chain of thoughts, and the final answer. Modalities involved include audio, images, video, 3D data, and tables; task types include question answering, grounding, and generation. This highlights MCoT’s adaptability and wide applicability.

read the caption

Figure 5: Examples of MCoT applications in various modalities and tasks.

🔼 This figure illustrates three main methodologies used in Multimodal Chain-of-Thought (MCoT) reasoning, categorized by how the rationale (the step-by-step reasoning process) is constructed. Prompt-based methods use carefully crafted prompts to guide the model’s reasoning. Plan-based methods allow models to dynamically explore and refine thoughts using tree, graph, or hypergraph structures. Learning-based methods incorporate rationale construction directly into the training or fine-tuning process, teaching the model to generate rationales alongside the final answers. The figure also highlights how each method integrates multimodal inputs from various modalities (Image, Video, Audio, etc.) into the LLM.

read the caption

Figure 6: MCoT reasoning methods under different rationale construction perspectives.

🔼 This figure illustrates different methodologies employed in MCoT reasoning, categorized by their structural reasoning approach. It shows how various methods handle the organization and flow of information during the reasoning process, contrasting asynchronous modality processing, defined procedures, and autonomous procedure learning. Asynchronous approaches process modalities independently, defined procedures follow predefined steps, and autonomous methods let the model decide the sequence of reasoning steps.

read the caption

Figure 7: MCoT methods under different structural reasoning perspectives.

🔼 Figure 8 illustrates different methodologies employed in MCOT reasoning, focusing on how enhancing the input information improves the reasoning process. It shows three main approaches: using expert tools to integrate external knowledge or perform specialized operations (e.g., geometric manipulation), retrieving world knowledge from external sources (e.g., knowledge graphs, databases) to enhance the reasoning process, and leveraging in-context knowledge retrieval from the existing information in the input or already generated rationales. Each approach is represented visually, highlighting how these techniques improve multimodal reasoning in MCOT.

read the caption

Figure 8: MCoT reasoning under perspectives with information enhancing.

🔼 Figure 9 illustrates different methodologies within Multimodal Chain-of-Thought (MCoT) reasoning, categorized by the granularity of their objectives. It shows how approaches vary depending on whether the goal is a high-level overview (coarse understanding), precise identification of specific elements (semantic grounding), or detailed analysis of individual components (fine-grained understanding). The figure visually represents the different levels of detail and information processing involved in each approach.

read the caption

Figure 9: MCoT reasoning under the perspectives of various objective granularities.

🔼 This figure illustrates the concept of Multimodal Chain-of-Thought (MCoT) reasoning using multimodal rationales. It shows that, unlike traditional text-only CoT reasoning, MCoT reasoning integrates multiple modalities (represented by ‘M’) into the rationale generation process. This means that the reasoning steps are not limited to text but can also incorporate information from images, audio, video, or other modalities, leading to a more comprehensive and nuanced understanding of the problem and ultimately, a more accurate answer.

read the caption

Figure 10: MCoT reasoning with multimodal rationale.

🔼 This figure illustrates different test-time scaling strategies used in Multimodal Chain-of-Thought (MCoT) reasoning. It shows how reinforcement learning (RL) can enhance the quality of reasoning and enable active long-CoT reasoning capabilities even without extensive annotated training data for long chains of thought. The figure depicts various approaches, emphasizing that supervised fine-tuning (SFT) is an optional component in this process. The options shown highlight different ways to scale the reasoning process at test time, thus allowing for more efficient and effective reasoning in resource-constrained settings.

read the caption

Figure 11: MCoT reasoning with test-time scaling strategies. RL can help improve reasoning quality, or active long-CoT reasoning ability without annotated long-CoT training data. SFT is optional.

🔼 This table compares various multimodal large language models (MLLMs) that utilize reinforcement learning for reasoning tasks. It highlights key characteristics such as the foundational LLM used, the modalities supported (text, image, video, audio), whether supervised fine-tuning (SFT) was employed, the reinforcement learning algorithm used, whether cold start capabilities were present and whether the model demonstrated an ‘aha moment’, an indication of insightful reasoning. Deepseek-R1, a text-only model, is included for comparative analysis.

read the caption

Table 2: Multimodal reasoning models utilizing reinforcement learning. Deepseek-R1 serves as a text-only LLM for comparison.

🔼 This table lists datasets and benchmarks used for training and evaluating Multimodal Chain-of-Thought (MCoT) reasoning models. It categorizes datasets based on whether they provide rationales (step-by-step explanations) during training. It also specifies the task (e.g., Visual Question Answering, Video Question Answering), domain (e.g., Science, Common Sense), modalities involved (Text, Image, Video, Audio), and the format of the answers (multiple-choice or open-ended). The table also indicates the number of samples in each dataset.

read the caption

Table 3: Datasets and Benchmarks for MCoT Training and Evaluation. “MC” and “Open” refer to multiple-choice and open-ended answer formats, while “T”, “I”, “V”, and “A” represent Text, Image, Video, and Audio, respectively.

🔼 This table presents a performance comparison of various multimodal large language models (MLLMs) from different institutions. The comparison is based on four benchmarks: MMMU (Validation set), MathVista (Mini version), Math-Vision, and EMMA (Mini version). Each benchmark likely tests different aspects of MLLM capabilities, such as mathematical reasoning, problem-solving, and understanding of visual and textual information. The table allows readers to quickly assess the relative performance of different MLLMs in these benchmark tasks, highlighting the strengths and weaknesses of various models.

read the caption

Table 4: Performance comparison of MLLMs from various institutions across four benchmarks: MMMU (Val), MathVista (Mini), Math-Vision, and EMMA (Mini).