Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning

🔼 This pie chart illustrates the hierarchical ontology developed for representing physical common sense within the Cosmos-Reason1 model. The ontology is structured into three main categories: Space, Time, and Fundamental Physics. Each of these high-level categories is further broken down into more specific subcategories (16 in total) which capture detailed aspects of physical understanding. For instance, the ‘Space’ category includes subcategories such as ‘Relationship’ (describing spatial relationships between objects), ‘Plausibility’ (assessing the feasibility of spatial arrangements), ‘Affordance’ (considering how objects can be used or interacted with), and ‘Environment’ (understanding the context of the surrounding area). Similarly, ‘Time’ is subdivided into concepts such as ‘Actions’, ‘Order’, ‘Causality’, and ‘Planning’, representing various temporal relationships and reasoning capabilities. The ‘Fundamental Physics’ category covers essential physical principles, including aspects of mechanics, electromagnetism, and thermodynamics, along with the crucial concept of ‘Object Permanence.’

read the caption

Figure 2: A pie chart showing our physical common sense ontology. The ontology has three categories (Space, Time, and Fundamental Physics) and 16 fine-grained subcategories.

🔼 The figure illustrates the architecture of the Cosmos-Reason1 multimodal large language model. The process begins with an input video and a text prompt. The video is first processed by a vision encoder, which extracts relevant visual features. These features are then projected into the LLM’s token embedding space using a projector, converting them into a format compatible with the text tokens from the prompt. The video tokens and text tokens are concatenated and fed into the core of the model: a hybrid Mamba-MLP-Transformer architecture. This architecture is designed for efficient and effective processing of long sequences of tokens, enabling the model to engage in long chain-of-thought reasoning processes to generate its output. The final output is a natural language response that incorporates the information from both the video and the text prompt.

read the caption

Figure 3: An illustration of our multimodal large language model architecture. Given an input video and an input text prompt, the video is projected into the LLM’s token embedding space as video tokens by a vision encoder followed by a projector. The text tokens are concatenated with the video tokens and fed into the LLM backbone, a hybrid Mamba-MLP-Transformer architecture. Our model can output responses with long chain-of-thought reasoning processes.

🔼 This figure details the architecture of the Cosmos-Reason1 models’ backbone, a hybrid Mamba-MLP-Transformer. The diagram shows how Transformer blocks (composed of self-attention and MLP layers) are combined with alternating Mamba-MLP modules for efficiency. The middle section illustrates the 8B parameter model, while the bottom shows the architecture of the larger 56B parameter model. The Mamba modules are designed to improve the efficiency of processing long sequences.

read the caption

Figure 4: An illustration of our hybrid Mamba-MLP-Transformer backbone architecture. The middle sub-figure represents the 8B LLM backbone, and the bottom sub-figure depicts the 56B LLM backbone. A Transformer block consists of a self-attention layer and an MLP layer. We also show an example of Alternating Mamba-MLP module on top of the figure.

🔼 This figure shows an overview of the Cosmos-Reason1 model architecture. The input is a video, processed through a vision encoder and a projector to align with the text embeddings of a pre-trained large language model (LLM). The model is trained in four stages: vision pre-training, general supervised fine-tuning (SFT), Physical AI SFT, and Physical AI reinforcement learning (RL). Two ontologies are used to represent physical common sense and embodied reasoning, and the model is evaluated on three benchmarks.

read the caption

(a)

🔼 This figure is a pie chart illustrating the hierarchical ontology for physical common sense used in the Cosmos-Reason1 model. The ontology categorizes fundamental knowledge about the physical world into three main categories: Space, Time, and Fundamental Physics. Each of these categories is further subdivided into more specific subcategories, representing fine-grained aspects of physical understanding. For example, ‘Space’ includes subcategories like ‘Relationship,’ ‘Plausibility,’ and ‘Affordance,’ reflecting different aspects of spatial reasoning. Similarly, ‘Time’ encompasses subcategories such as ‘Actions,’ ‘Order,’ and ‘Causality,’ while ‘Fundamental Physics’ incorporates concepts like ‘Mechanics,’ ‘Electromagnetism,’ and ‘Thermodynamics.’

read the caption

(b)

🔼 This figure shows example video frames from the Physical AI Supervised Fine-Tuning datasets. The images depict various scenarios involving different agents (humans, robots) performing actions in real-world environments. These examples illustrate the diversity and complexity of the data used to train the Cosmos-Reason1 models on physical common sense and embodied reasoning.

read the caption

(c)

🔼 This figure shows an example of video frames from the Physical AI Supervised Fine-tuning datasets. The images depict various scenarios involving different embodied agents (such as robots and humans) performing different tasks in real-world environments. These videos are used in the Physical AI SFT stage of training the Cosmos-Reason1 model to help it learn physical common sense and develop embodied reasoning capabilities.

read the caption

(d)

🔼 This figure shows an example of video frames from the Physical AI Supervised Fine-tuning datasets. It highlights the diversity of scenarios and tasks included in the dataset, encompassing various activities and agent types relevant to physical AI, such as robotic manipulation and autonomous driving. The images represent a small subset of the larger dataset used to train the models.

read the caption

(e)

🔼 This figure shows an example of video frames from the Physical AI Supervised Fine-Tuning datasets. These datasets were used to enhance the model’s ability to understand Physical AI-specific tasks, focusing on embodied reasoning capabilities such as understanding the spatial relationships between objects, the temporal order of events, and object permanence. The images illustrate the visual diversity captured within the data, showcasing a variety of actions and contexts that the model must reason about to successfully complete Physical AI tasks.

read the caption

(f)

🔼 Figure 5 presents example video frames from the Physical AI Supervised Fine-Tuning stage of the Cosmos-Reason1 model training. These diverse video clips showcase various scenarios used to train the model in understanding and reasoning about physical phenomena. The visual examples highlight the real-world contexts and diverse physical actions the model is trained on, demonstrating the wide range of physical situations represented in the dataset.

read the caption

Figure 5: Example of video frames from our Physical AI supervised fine-tuning datasets.

🔼 Figure 6 illustrates the data curation pipeline for embodied reasoning within the context of Physical AI. The example focuses on the AgiBot dataset. The process begins by extracting short video segments that represent individual subtasks from a longer video. These clips are then annotated with captions that provide detailed context about the state of the scene and the actions taken by the agent. Next, question-answer pairs are generated, focusing on predicting the most probable next subtask. The question and caption are provided as input to the DeepSeek-R1 model, which then generates a reasoning trace (a chain of thought) to answer the question. Finally, this reasoning trace is cleaned and rewritten to create high-quality supervised fine-tuning (SFT) samples.

read the caption

Figure 6: Embodied reasoning SFT data curation pipeline. We demonstrate an illustrative example for AgiBot, where we (1) extract short horizon segments corresponding to the subtask, (2) caption the extracted clip to obtain state-action context, (3) curate QA pairs for “next plausible subtask prediction”, (4) prompt R1 with the question and caption to elicit reasoning, (5) clean and rewrite the reasoning trace to obtain valid SFT samples.

🔼 This table presents an ontology for embodied reasoning in Physical AI systems. It categorizes key capabilities essential for embodied reasoning (processing complex sensory inputs, predicting action effects, respecting physical constraints, and learning from interactions) and illustrates them with examples across various agent types (natural agents like humans and animals, and robotic systems like robot arms, humanoid robots, and autonomous vehicles). Each cell shows a specific example of how the capability is demonstrated by the corresponding agent type.

read the caption

Table 2: Embodied reasoning ontology, with an example for each combination of capability and agent type.

🔼 This table presents a detailed comparison of the architectural configurations for the two Cosmos-Reason1 models: Cosmos-Reason1-8B and Cosmos-Reason1-56B. It breaks down the specifications of each model’s vision encoder, projector, and LLM backbone. Specifically, it lists the architecture type, input and output dimensions, number of layers, hidden dimensions, and other relevant hyperparameters for each component. This allows for a comprehensive understanding of the differences in model size and complexity between the two variants.

read the caption

Table 3: Configuration details of Cosmos-Reason1 models.

🔼 This table presents a summary of the datasets used for training the Physical AI Supervised Fine-Tuning stage of the Cosmos-Reason1 model. It breaks down the data by category (Physical Common Sense VQA, Embodied Reasoning, and Intuitive Physics), indicating the number of samples (free-form and multiple-choice questions) used for both understanding and reasoning annotations. It also specifies the sources from which data is obtained (e.g. BridgeData V2, RoboVQA, etc.). This detailed breakdown helps understand the composition and scale of data used in training the model to reason about Physical AI.

read the caption

Table 4: A summary of datasets used for physical AI supervised fine-tuning.

🔼 This table presents the datasets used for the Physical AI Reinforcement Learning phase of the Cosmos-Reason1 model training. It breaks down the number of samples used for training in several categories: Physical Common Sense (MCQs and free-form questions), Embodied Reasoning (data from various sources like BridgeData V2, RoboVQA, etc.), and Intuitive Physics (data for spatial puzzles, arrow-of-time, and object permanence). The numbers represent the count of samples prepared for reinforcement learning in each category. This data is crucial in understanding the composition of the training data for this stage of model development, showing how different types of physical reasoning tasks were weighted.

read the caption

Table 5: Datasets for Physical AI reinforcement learning post-training.

🔼 This table presents a statistical overview of the benchmarks used in the paper to evaluate the performance of the Cosmos-Reason1 models. It breaks down the number of questions and videos included in both the physical common sense and embodied reasoning benchmark sets. For common sense, it shows the distribution of questions across Space, Time, and Fundamental Physics categories. The embodied reasoning section details the number of questions and videos across various datasets and agent types (robot arms, autonomous vehicles, etc.).

read the caption

Table 6: Statistics of our curated benchmarks.

🔼 This table presents the results of evaluating various models on a physical common sense benchmark. It compares the performance of Cosmos-Reason1-8B and Cosmos-Reason1-56B to several other leading language models (OpenAI 01, Qwen2.5-VL-7B, Qwen2.5-VL-72B, and Gemini 2.0 Flash) across three sub-benchmarks focusing on different aspects of physical common sense: Space, Time, and Fundamental Physics. The table shows the accuracy of each model on each sub-benchmark, demonstrating the effectiveness of the Cosmos-Reason1 models, particularly the 56B variant, in this domain.

read the caption

Table 7: Evaluation on physical common sense benchmark.

🔼 This table presents the performance comparison of different models on an embodied reasoning benchmark. The benchmark evaluates models’ ability to perform tasks requiring physical reasoning across different agent types (humans, robot arms, humanoid robots, autonomous vehicles). Each model’s accuracy is reported for six different sub-benchmarks (BridgeData V2, RoboVQA, Agibot, HoloAssist, AV, RoboFail), providing a comprehensive evaluation of their embodied reasoning capabilities. The average accuracy across all six sub-benchmarks is also included for each model.

read the caption

Table 8: Evaluation on embodied reasoning benchmark.

🔼 This table presents the results of evaluating the Cosmos-Reason1 models (8B and 56B) and several other models on benchmarks designed to assess physical common sense and embodied reasoning capabilities. It shows the accuracy of each model on various sub-tasks within those benchmarks, providing a comparison of the performance of Cosmos-Reason1 to other state-of-the-art models.

read the caption

Table 9: Evaluation on physical common sense and embodied reasoning benchmark.

🔼 This table presents the results of evaluating various models on three intuitive physics tasks: Arrow of Time, Spatial Puzzles, and Object Permanence. The evaluation measures each model’s ability to reason about the direction of time in a video, solve spatial puzzles by identifying correctly ordered image patches, and reason about object permanence in the context of occlusions. The table compares the performance of several existing LLMs and VLMs against Cosmos-Reason1, both before and after RL fine-tuning, highlighting the impact of the model’s training on these fundamental physical reasoning abilities.

read the caption

Table 10: Evaluation on intuitive physics benchmark.