M3: 3D-Spatial MultiModal Memory

🔼 The figure illustrates the architecture of M3, a 3D Spatial Multimodal Memory system. It shows how the system combines multiple foundation models (e.g., CLIP, LLaMA, DINO) to extract multi-granular semantic knowledge from a scene represented as a video (V). This knowledge, along with the scene’s spatial structure captured using 3D Gaussian splatting, is integrated into a unified multimodal memory representation. This memory allows for various downstream tasks, including image retrieval, caption generation, and object grounding. The figure visually depicts the process, showing how foundation models process the scene’s input video to generate features and how Gaussian splatting structures this information for efficient storage and retrieval.

read the caption

Figure 2: A scene (𝐕𝐕\mathbf{V}bold_V) is composed of both structure (𝐒𝐒\mathbf{S}bold_S) and knowledge (𝐈𝐈\mathbf{I}bold_I). To model these, we leverage multiple foundation models to extract multi-granularity scene knowledge, and employ 3D Gaussian splatting to represent the spatial structure. By combining these techniques, we construct a spatial multimodal memory (M3), which enables downstream applications such as retrieval, captioning and grounding.

🔼 Figure 3 illustrates the MultiModal Memory (M3) pipeline. A video sequence is input, and foundation models extract raw features. These high-dimensional features are then reduced using Algorithm 1 to create principal scene components (PSCs), which are stored in a memory bank. Optimizable attribute queries are applied to Gaussian primitives, and a Gaussian Memory Attention mechanism generates final rendered features. These rendered features retain the semantic information from the foundation models and are used for downstream tasks via connections to various model heads.

read the caption

Figure 3: Given a video sequence, we utilize foundation models (𝐅𝐅\mathbf{F}bold_F) to extract raw features (𝐑𝐑\mathbf{R}bold_R). These features are reduced using Algorithm 1, producing principal scene components (𝐏𝐒𝐂𝐏𝐒𝐂\mathbf{PSC}bold_PSC), which are stored in a memory bank. We introduce optimizable attribute queries (q𝑞qitalic_q) to Gaussian primitives, and apply a Gaussian Memory Attention (𝐀g⁢msubscript𝐀𝑔𝑚\mathbf{A}_{gm}bold_A start_POSTSUBSCRIPT italic_g italic_m end_POSTSUBSCRIPT) mechanism to produce the final rendered features (𝐑^^𝐑\hat{\mathbf{R}}over^ start_ARG bold_R end_ARG), which can be linked back to various heads of the foundation models.

🔼 This UMAP visualization shows the embedding manifolds for different foundation models used in the paper. Each model’s embeddings form a distinct cluster shape, illustrating how each model focuses on different aspects of scene understanding. The different shapes highlight the unique features and perspectives extracted by the diverse foundation models, which range from vision-language models to perception models. This visualization emphasizes the multi-modality and multi-granularity of features extracted for the proposed multimodal memory.

read the caption

Figure 4: The UMAP visualization of model embedding manifolds reveals distinct shapes, reflecting different focus.

🔼 This figure illustrates the process of extracting patch-level visual embeddings from various foundation models for downstream tasks. It shows how different models (DINOv2, CLIP, SigLIP, LLaMA3, LLaMAv) process input images, focusing on their respective strengths in vision and language understanding. The extracted features from different layers of these models are used to build a comprehensive understanding of the scene at a granular level. The output shows how the combined features can be effectively used in applications such as visual question answering, captioning, and retrieval.

read the caption

Figure 5: Illustration of patch-level visual embedding extraction their applications.

🔼 Figure 6 presents a qualitative comparison of the M3 model’s performance across four diverse datasets: Garden, Playroom, Drjohnson, and Tabletop. The images showcase the model’s consistent ability to accurately capture and render fine-grained details, intricate textures, and complex scene structures, regardless of the dataset’s specific characteristics. The results demonstrate M3’s robustness and generalizability in handling various visual environments.

read the caption

Figure 6: Qualitative results across datasets using M3. The figure showcases the consistent performance of the M3 across various datasets (Garden, Playroom, Drjohnson, Table-top).

🔼 The figure shows a quadruped robot grasping a rubber duck. The robot uses the M3 model to locate and identify the object based on a text query (‘yellow bath duck’). The image sequence demonstrates the robot’s ability to identify the object, determine its position using a depth camera, and execute a grasping action. This showcases the real-world applicability of the M3 model for tasks involving object manipulation.

read the caption

Figure 7: Real robot deployment.

🔼 This figure showcases the capabilities of the M3 model in real-world robot manipulation tasks. Three scenarios are depicted: part-level understanding (picking up a specific object by its handle), multi-scene understanding (grounding and grasping objects across different environments), and long-horizon tasks (completing a multi-step task such as retrieving an object from a different location and placing it elsewhere). Each scenario is demonstrated with a sequence of images illustrating the robot’s actions.

read the caption

Figure 8: Real robot deployment on part-level understanding, multi-scene and long-horizon tasks.

🔼 This table presents a comprehensive quantitative evaluation of the M3 model’s performance across various foundation models and scenes. It shows feature and RGB metrics (mIoU, cIoU, AP50, AP60, I2T@1, I2T@5, I2T@10, T2I@1, T2I@5, T2I@10) for different scenes (Train, Playroom, Geisel) and the ground truth. This allows for a comparison of M3’s performance against the ground truth and other methods. The metrics assess both the similarity between features extracted by M3 and the ground truth, as well as the model’s performance on downstream tasks, providing insights into the effectiveness of M3 in representing and recalling scene information.

read the caption

Table 2: Feature/RGB metrics for all foundation models and scene.

🔼 This table presents an ablation study analyzing the impact of using different numbers of foundation models within the M3 model. It shows how the performance changes as more models (CLIP, SigLIP, DINOv2, SEEM, LLaMA3, and LLaMAv) are incorporated into the system. The results are evaluated using PSNR, cosine similarity, and L2 distance metrics across different feature spaces, showing the trade-off between model complexity and performance.

read the caption

Table 3: Ablation on the number of foundation models in M3.

🔼 This table presents an ablation study analyzing the impact of varying the dimensionality and distillation techniques for each foundation model used in the M3 system. It shows how different settings of feature vector dimensions and the application (or not) of feature distillation affect performance across various metrics and foundation models. This helps to understand the optimal configuration for balancing model complexity and performance.

read the caption

Table 4: Ablation on the dimensions and distillation for each foundation model.