MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs

🔼 This figure shows a sample from the Cubify Anything VQA (CA-VQA) dataset. It illustrates the multiple data modalities included in the dataset for each image. A main reference image is shown, along with up to four additional support frames from slightly different viewpoints. Each frame (both reference and support) provides three depth maps: one from a high-accuracy FARO laser scanner (ground truth), one from Apple’s ARKit (LiDAR-fused), and one from a monocular depth estimation model (DepthPro). For each support frame, relative pose information (showing its position and orientation relative to the reference frame) and camera intrinsic parameters are also included.

read the caption

Figure 2: CA-VQA Data Example. Example of a single sample from our dataset. Each reference frame has between 0-4 multi-view support frames. All frames (reference and support) come with three metric depth maps: Ground truth (FARO laser), ARKit Depth (LiDAR-fused) and Monocular (DepthPro). Each support frame contains the relative pose from the reference image, along with camera intrinsics.

🔼 This figure illustrates the process of using depth information to answer spatial questions. The model first identifies objects in an image and determines their 2D bounding boxes. Then, it queries a ’tool’ (a function that extracts depth information) for the median depth value within each bounding box. This depth information is then used by the model in a chain-of-thought reasoning process to answer a question involving spatial relationships, such as ‘Is the pillow behind the television?’

read the caption

Figure 3: Example of leveraging depth maps via tool-use. The model predicts the objects’ 2D bounding boxes and function calls, receives the tool outputs (which is the median depth value within the box, marked with an ×\mathbf{\times}×), and finally reasons about the answer.

🔼 Figure 4 presents a qualitative comparison of different models’ performance on a complex spatial reasoning task from the CA-VQA benchmark. It highlights the superior performance of the MM-Spatial model. The figure shows that while strong commercial and research models fail to accurately answer the question, MM-Spatial provides a much better response. Further improvements are observed when using Chain-of-Thought (CoT) reasoning and ground truth depth, demonstrating the model’s ability to ground objects in 2D space, estimate depth accurately, and reason spatially. The use of monocular depth estimation, while also helpful, is shown to be less accurate than ground truth depth.

read the caption

Figure 4: Qualitative Example. We show the predictions of various models on a challenging example from our CA-VQA benchmark. Strong commercial (2a&b) and research models (2c&d) fail. MM-Spatial (1a) is much better, and even more so with CoT enabled (1b), demonstrating our model’s strong object grounding (see predicted 2D boxes in the image), depth estimation, and spatial reasoning ability. Accuracy improves further when leveraging ground-truth depth via tool-use (1c), although our CoT model’s (1b) predictions are very close to that, for both the intermediate depth values and final answer; monocular estimated depth (1d) is less accurate and yields a worse result.

🔼 Figure 5 showcases example question-answer pairs from the Cubify Anything VQA (CA-VQA) dataset. CA-VQA is designed to improve 3D spatial reasoning capabilities in multimodal large language models (MLLMs). It leverages high-quality 3D ground truth annotations from the CA-1M dataset to create diverse spatial perception questions. These questions cover a wide range of tasks including relative spatial relationships between objects, metric measurements (distances and sizes), and 3D object bounding box identification. The figure visually demonstrates the variety of question types and the dataset’s focus on detailed 3D spatial understanding.

read the caption

Figure 5: CA-VQA Overview. Example QA pairs from our Cubify Anything VQA (CA-VQA) dataset, aiming to unlock object-centric 3D spatial understanding in MLLMs. Using high-quality 3D ground truth annotations from CA-1M [61], we generate spatial perception questions across a variety of different tasks, e.g., involving relative relationships, metric measurements, and 3D object bounding boxes.

🔼 Figure 6 presents example questions and answers from the CA-VQA dataset, categorized into Binary, Counting, and Multi-choice question types. The Binary examples showcase questions about spatial relationships (e.g., Is object A in front of object B?), relative object sizes, and object presence. Counting questions involve counting the number of objects of a specific type in the image. Multi-choice examples combine different question types into a multiple-choice format. The figure visually demonstrates the variety of question formats and types of spatial reasoning tasks included in the CA-VQA dataset.

read the caption

Figure 6: Examples of CA-VQA data samples from the Binary, Counting and Multi-choice categories.

🔼 Figure 7 shows example questions and answers from the CA-VQA dataset that belong to two categories: Regression (Metric Estimation) and 2D Grounding. The Regression examples demonstrate questions that require estimating distances (e.g., how far away is an object, distance between two objects) and sizes (e.g., height, length) of objects. The 2D Grounding examples showcase questions that ask for the 2D image coordinates of objects given either their names or material properties. The figure illustrates the diversity of questions and the types of answers expected in CA-VQA.

read the caption

Figure 7: Examples of CA-VQA data samples from the Regression (Metric Estimation) and 2D Grounding categories.

🔼 Table 2 presents a performance comparison between the MM-Spatial model and the MM1.5 baseline model across various benchmark categories. It highlights that MM-Spatial significantly improves performance in the ‘Spatial’ category while maintaining competitive performance with MM1.5 in other categories such as General, Knowledge, Text-rich, and Referring/Grounding. This demonstrates MM-Spatial’s effectiveness as a generalist multimodal large language model (MLLM) that excels in spatial reasoning without sacrificing performance on other tasks.

read the caption

Table 2: Benchmark Category Results MM-Spatial is a generalist MLLM that improves strongly on the Spatial category while rivaling the MM1.5 baseline across the other task categories.

🔼 Table 3 presents a comprehensive evaluation of the MM-Spatial model on the Cubify Anything VQA (CA-VQA) benchmark. It compares MM-Spatial’s performance across various spatial reasoning tasks (binary classification, counting, multi-choice questions, and metric regression) against several leading open-source and commercial large language models. The results show that MM-Spatial-3B significantly surpasses these other models, highlighting its superior 3D spatial understanding capabilities. The table also demonstrates the positive impact of incorporating additional input signals, such as multi-view images and depth information (obtained through both tool-use and Chain-of-Thought reasoning), further enhancing the accuracy of MM-Spatial’s predictions. The improvement through chain-of-thought highlights the model’s ability to accurately estimate metric depth, a critical component in advanced spatial reasoning.

read the caption

Table 3: CA-VQA Results. MM-Spatial-3B significantly outperforms (much larger) top open-source and commercial models across all tasks, demonstrating its strong spatial understanding ability. Model performance is further improved by incorporating multi-view and/or depth as additional input signals, as well as by leveraging CoT, which relies on our model’s ability to accurately estimate metric depth.

🔼 Table 4 presents a comprehensive comparison of MM-Spatial-3B’s performance on the CV-Bench benchmark against other state-of-the-art (SOTA) models. CV-Bench evaluates performance across both 2D and 3D spatial reasoning tasks. The results demonstrate that MM-Spatial-3B significantly surpasses the performance of larger SOTA models, particularly in 3D spatial reasoning tasks. Furthermore, incorporating Chain-of-Thought (CoT) reasoning and utilizing depth input (either via sensor or monocular depth estimation) further improves the performance of MM-Spatial-3B. Notably, MM-Spatial-3B achieves near-perfect accuracy on indoor 3D tasks and also shows high performance on outdoor 3D tasks, showcasing its ability to generalize well beyond the training data.

read the caption

Table 4: CV-Bench Results. MM-Spatial-3B substantially outperforms the (much larger) SOTA models, with CoT and depth input further improving performance. It almost fully solves the indoor 3D tasks, while also excelling at the out-of-domain outdoor 3D tasks.

🔼 This table presents a quantitative evaluation of metric depth estimation. It compares the performance of the MM-Spatial model’s Chain-of-Thought (CoT) approach against two baselines: a monocular depth estimation model (DepthPro) and a LiDAR-based depth estimation method (ARKit Depth). The evaluation uses the CA-VQA benchmark and focuses on the accuracy of depth prediction, measured using two common metrics: δ1 (accuracy at 25% relative error) and AbsRel (absolute relative error). The results show that the MM-Spatial + CoT model surpasses DepthPro’s performance, although the LiDAR-based ARKit Depth remains the most accurate.

read the caption

Table 5: Metric Depth Estimation Results. We evaluate the metric depth estimates of our CoT model produced as part of its responses on the CA-VQA benchmark. We compare against the tool-use estimates based on Monocular (DepthPro [12]) and ARKit Depth, i.e., the median depth value within the 2D box predicted by MM-Spatial + CoT (generalist & specialist). We report the δ1subscript𝛿1\delta_{1}italic_δ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT (accuracy at 25% relative error) and AbsRel (absolute relative error) metrics [57] commonly used in the depth estimation literature [12], computed against GT FARO depth. MM-Spatial + CoT outperforms DepthPro. LiDAR-derived ARKit Depth is best.

🔼 Table 6 presents a comparison of the performance of different models on the SpatialRGPT-Bench benchmark. The key finding is that MM-Spatial-3B achieves state-of-the-art (SOTA) results, surpassing even the SpatialRGPT-VILA-1.5-8B model, which uses full depth encoding. The table shows that MM-Spatial-3B’s performance improves further when using tool-use monocular depth. Additionally, it demonstrates that training MM-Spatial-3B on a combined dataset consisting of both CA-VQA* and OSD yields the best overall results.

read the caption

Table 6: SpatialRGPT-Bench Results. MM-Spatial-3B achieves SOTA with both image-only input and tool-use monocular depth, outperforming SpatialRGPT-VILA-1.5-8B (which fully encodes depth). Training on a mixture of CA-VQA⋆ and OSD performs best.

🔼 This table presents the results of experiments evaluating different ratios of training data for a multimodal large language model (MLLM) specialized for 3D spatial understanding. The model, MM-Spatial, was trained on a mixture of general-purpose data and spatial data from the CA-VQA dataset. The table compares performance across various benchmark categories (General, Knowledge, Text-rich, Referring & Grounding, and Spatial) for different ratios of spatial to general data in the training set (e.g., 0:1, 1:1, 2:1, 4:1, 8:1, 1:0). It shows that a 2:1 ratio provides a good balance between performance on the spatial category and maintaining performance on other categories, confirming the model’s generalist nature. The last row shows a control experiment training a specialized spatial model, illustrating that a generalist approach with mixed data is superior to a specialist approach using only CA-VQA data.

read the caption

Table 7: Data Mixture Ratio Results. Comparison of different data mixture ratios – both (Rel)ative to the General category (as in MM1.5), and (Eff)ective when considering the dataset sizes – on aggregated metrics across the different benchmark categories. Overall, MM-Spatial is a generalist MMLM that improves a lot on the Spatial category while maintaining strong performance on the other categories. The data mixture ratio of 2:1 (spatial:general) provides a good performance trade-off and is used for MM-Spatial throughout. The last line considers a spatial Specialist Model that is trained on CA-VQA only; this model provides only a minor improvement on the spatial category, while regressing substantially on all other benchmark categories.

🔼 Table 8 presents a comparison of the MM-Spatial model’s performance against state-of-the-art (SOTA) models on a range of knowledge and general benchmark tasks. It shows scores for various sub-tasks within each benchmark, providing a comprehensive evaluation of the model’s capabilities beyond just spatial reasoning. Note that some SOTA model scores are sourced from a different reference paper ([33]). The table highlights the model’s ability to perform competitively on these general tasks while excelling in spatial understanding.

read the caption

Table 8: Knowledge and General Benchmark Results. Comparison with SOTA models on knowledge and general benchmarks. (††\dagger†) Sum of P and C scores. Gemini-1.5-Pro, GPT-4V and GPT-4o numbers are from [33].

🔼 This table presents a comparison of the performance of various models, including the proposed MM-Spatial model, on several text-rich benchmarks. The benchmarks assess the models’ abilities to understand and generate text in the context of rich textual data. The results are reported as scores and allow for a quantitative comparison of the different models’ capabilities in handling complex textual information.

read the caption

Table 9: Text-rich Benchmark Results. Comparison with SOTA models on text-rich benchmarks. (††\dagger†) Numbers are obtained from [63].

🔼 This table compares the performance of the MM-Spatial model against state-of-the-art (SOTA) models on several established 2D referring and grounding benchmarks. It evaluates the model’s ability to accurately locate and identify objects within images using natural language descriptions. The benchmarks assess different aspects of this capability, and the table provides a quantitative comparison of results (e.g., accuracy scores) across these benchmarks.

read the caption

Table 10: 2D Referring & Grounding Benchmark Results. Comparison with SOTA models on 2D referring and grounding benchmarks.

🔼 Table 11 analyzes the impact of a blind filtering technique on the CA-VQA benchmark. This technique removes questions easily solvable without visual input, thus increasing the benchmark’s reliance on visual reasoning. The table compares the performance of GPT-4, GPT-4V, and MM-Spatial models before and after applying the filter, both with and without visual input. The results show that after filtering, blind models (no visual input) perform significantly worse, while the improvement in performance gained by using visual input increases substantially for all models. This demonstrates that the filtering effectively reduces bias from language priors and enhances the benchmark’s ability to test true visual understanding.

read the caption

Table 11: CA-VQA Blind Filtering Analysis. We study how the improvement from using vision (i.e., comparing a vision-evaluated model vs. a blind-evaluated model) changes after applying the blind filtering strategy outlined in Sec. 3.1, which follows [25]. Our results confirm that after applying our filtering strategy, 1) blind models perform substantially worse, and 2) vision improvements (i.e., the delta between vision and blind models) increase substantially, for both GPT-4/V and MM-Spatial. This highlights the effectiveness of our blind filtering procedure in ensuring that our CA-VQA benchmark becomes more reliant on vision input (i.e., less susceptible to a strong language prior).