CrossOver: 3D Scene Cross-Modal Alignment

🔼 CrossOver is a multimodal scene understanding framework that aligns different modalities (RGB images, point clouds, CAD models, floor plans, and text descriptions) within a shared embedding space. It uses a two-stage approach: first, an Instance-Level Multimodal Interaction module captures interactions between modalities at the instance level; second, a Scene-Level Multimodal Interaction module combines instance-level information to generate a single scene-level feature vector. Finally, Unified Dimensionality Encoders process each modality independently, interacting with the scene-level feature vector, to eliminate the need for precise semantic instance information and enable cross-modal alignment at the scene level.

read the caption

Figure 2: Overview of CrossOver. Given a scene 𝒮𝒮\mathcal{S}caligraphic_S and its instances 𝒪isubscript𝒪𝑖\mathcal{O}_{i}caligraphic_O start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT represented across different modalities ℐ,𝒫,ℳ,ℛ,ℱℐ𝒫ℳℛℱ\mathcal{I},\mathcal{P},\mathcal{M},\mathcal{R},\mathcal{F}caligraphic_I , caligraphic_P , caligraphic_M , caligraphic_R , caligraphic_F, the goal is to align all modalities within a shared embedding space. The Instance-Level Multimodal Interaction module captures modality interactions at the instance level within the context of a scene. This is further enhanced by the Scene-Level Multimodal Interaction module, which jointly processes all instances to represent the scene with a single feature vector ℱ𝒮subscriptℱ𝒮\mathcal{F_{S}}caligraphic_F start_POSTSUBSCRIPT caligraphic_S end_POSTSUBSCRIPT. The Unified Dimensionality Encoders eliminate dependency on precise semantic instance information by learning to process each scene modality independently while interacting with ℱ𝒮subscriptℱ𝒮\mathcal{F_{S}}caligraphic_F start_POSTSUBSCRIPT caligraphic_S end_POSTSUBSCRIPT.

🔼 This figure illustrates the inference pipeline for cross-modal scene retrieval. It begins with a query scene represented by a specific modality (e.g., a point cloud). A dimensionality-specific encoder processes this query, generating a feature vector in a shared embedding space. This feature vector is then compared to feature vectors of scenes in the target modality (e.g., floorplans) stored in a database. The scene with the closest feature vector in the target modality is identified as the retrieved scene.

read the caption

Figure 3: Cross-modal Scene Retrieval Inference Pipeline. Given a query modality (𝒫𝒫\mathcal{P}caligraphic_P) that represents a scene, we obtain with the corresponding dimensionality encoder its feature vector (ℱ3⁢Dsubscriptℱ3𝐷\mathcal{F}_{3D}caligraphic_F start_POSTSUBSCRIPT 3 italic_D end_POSTSUBSCRIPT) in the shared cross-modal embedding space. We identify the closest feature vector (ℱ2⁢Dsubscriptℱ2𝐷\mathcal{F}_{2D}caligraphic_F start_POSTSUBSCRIPT 2 italic_D end_POSTSUBSCRIPT) in the target modality (ℱℱ\mathcal{F}caligraphic_F) and retrieve the corresponding scene from a database of scenes in ℱℱ\mathcal{F}caligraphic_F.

🔼 This figure shows the performance of CrossOver and other multi-modal methods on instance-level retrieval tasks using the ScanNet dataset. Specifically, it displays the recall (the percentage of correctly identified instances) for various modality pairs (e.g., image to text, point cloud to mesh). Each bar represents a different modality pair, and the height of the bar indicates the recall rate. The results showcase CrossOver’s ability to achieve high recall across diverse modality combinations, demonstrating its effectiveness in cross-modal instance retrieval. The chart visually compares CrossOver’s performance against other state-of-the-art multi-modal methods, highlighting CrossOver’s superior performance.

read the caption

(a) Instance Matching Recall on ScanNet

🔼 This figure presents a comparison of scene-level matching recall achieved by different methods on the ScanNet and 3RScan datasets. Scene-level matching recall assesses the ability of a method to retrieve the correct scene from a database given a query scene represented by a specific modality (e.g., RGB images or point clouds). The figure illustrates the superior performance of the proposed CrossOver method compared to several baselines, highlighting its robustness across different datasets and modalities.

read the caption

(b) Scene-Level Matching Recall on ScanNet and 3RScan

🔼 Figure 4 presents the results of cross-modal instance retrieval experiments conducted on the ScanNet and 3RScan datasets. Part (a) compares the performance of CrossOver to a simpler ‘Instance Baseline’ which doesn’t leverage scene-level context. This demonstrates that CrossOver’s superior performance stems from its ability to learn relationships between modalities at the scene level, not just the object level. Although CrossOver doesn’t explicitly train on every possible modality combination, it shows emergent cross-modal behavior. Part (b) shows that CrossOver significantly outperforms other existing multi-modal methods (ULIP-2 and PointBind) on both datasets, highlighting the robustness and effectiveness of its learned cross-modal interactions.

read the caption

Figure 4: Cross-Modal Instance Retrieval on ScanNet and 3RScan. (a) Even though CrossOver does not explicitly train all modality combinations, it achieves emergent behavior within the embedding space. The same applies to our Instance Baseline (Ours). CrossOver performs better than our self-baseline since it incorporates more scene context in the fusion of modalities. (b) Our method outperforms all baselines in all datasets, showcasing the robustness of learned cross-modal interactions.

🔼 This figure demonstrates the effectiveness of the CrossOver model in cross-modal scene retrieval. Using a floorplan as input (query modality ℱ), the model successfully retrieves the corresponding point cloud scene (target modality 𝒫) as the top match. In contrast, PointBind and a simpler baseline model fail to identify the correct scene among the top four results. The visualization also highlights a noteworthy characteristic of the CrossOver embedding space: temporal scenes (i.e., the same scene captured at different times) cluster closely together. This is evident in the examples where similar scene layouts are retrieved with slightly different viewpoints (k=2 and k=3), demonstrating the model’s ability to capture and maintain scene-level consistency across time.

read the caption

Figure 5: Cross-Modal Scene Retrieval Qualitative Results on ScanNet. Given a scene in query modality ℱℱ\mathcal{F}caligraphic_F, we aim to retrieve the same scene in target modality 𝒫𝒫\mathcal{P}caligraphic_P. While PointBind and the Instance Baseline do not retrieve the correct scene within the top-4 matches, CrossOver identifies it as the top-1 match. Notably, temporal scenes appear close together in CrossOver’s embedding space (e.g., k=2𝑘2k=2italic_k = 2, k=3𝑘3k=3italic_k = 3), with retrieved scenes featuring similar object layouts to the query scene, such as the red couch in k=4𝑘4k=4italic_k = 4.

🔼 Figure 6 presents a comparative analysis of cross-modal scene retrieval performance using different methods. The results are shown in plots for top 1, 5, 10, and 20 scene matching recall. Three modality pairs are compared: images to point clouds (I→P), images to text descriptions (I→R), and point clouds to text descriptions (P→R). The ‘Ours’ and ‘Instance Baseline’ methods were not explicitly trained on the P→R pair. The experiment used a database of 306 scenes. The plots demonstrate that the ‘Ours’ method significantly outperforms other methods, particularly the ‘Instance Baseline’, highlighting the improved cross-modal scene interactions enabled by the unified encoders used in the ‘Ours’ approach.

read the caption

Figure 6: Cross-Modal Scene Retrieval on ScanNet (Scene Matching Recall). Plots showcase the top 1, 5, 10, 20 scene matching recall of different methods on three modality pairs: ℐ→𝒫→ℐ𝒫\mathcal{I}\rightarrow\mathcal{P}caligraphic_I → caligraphic_P, ℐ→ℛ→ℐℛ\mathcal{I}\rightarrow\mathcal{R}caligraphic_I → caligraphic_R, 𝒫→ℛ→𝒫ℛ\mathcal{P}\rightarrow\mathcal{R}caligraphic_P → caligraphic_R. Ours and the Instance Baseline have not been explicitly trained on 𝒫→ℛ→𝒫ℛ\mathcal{P}\rightarrow\mathcal{R}caligraphic_P → caligraphic_R. Results are computed on a database of 306 scenes and showcase the superior performance of our approach. Once again, the difference between Ours and our self-baseline is attributed to the enhanced cross-modal scene-level interactions achieved with the unified encoders.

🔼 This figure presents a comparison of instance matching recall across different modalities. The results are shown for various modality pairs, including image-to-point cloud, image-to-mesh, point cloud-to-mesh, and image-to-text. The figure showcases the relative performance of the proposed CrossOver model compared to several baseline methods, highlighting its ability to achieve robust performance across various modality combinations even without explicit object-level alignment.

read the caption

(a) Instance Matching Recall

🔼 This figure shows the scene-level matching recall results on the ScanNet and 3RScan datasets. It compares the performance of the proposed CrossOver method with several baselines, including ULIP-2, PointBind, and an instance baseline. The results are presented for different recall thresholds (R@25%, R@50%, R@75%), which represent the percentage of correctly retrieved scenes among the top 25%, 50%, and 75% of retrieved results. The figure visually demonstrates the superior performance of CrossOver across all datasets and metrics.

read the caption

(b) Scene-Level Matching Recall

🔼 Figure 7 shows the results of training the CrossOver model on different combinations of the ScanNet and 3RScan datasets. The graph displays the improvement in both instance-level and scene-level matching recall when training on both datasets compared to training on only one. The improvement highlights the benefit of using a larger and more diverse dataset for training. It also shows that training only on the 3RScan dataset and testing on ScanNet results in significantly lower performance, as expected, since 3RScan has different modalities (RGB images, point clouds, and text descriptions) than ScanNet. This demonstrates the model’s ability to leverage data from multiple sources to improve overall performance.

read the caption

Figure 7: Scaled-up training performance on ScanNet. When training on both ScanNet and 3RScan datasets together, results improve from any individual dataset training. As expected, training on 3RScan and evaluating on ScanNet will have limited performance. Note that the 3RScan includes only the ℐℐ\mathcal{I}caligraphic_I, 𝒫𝒫\mathcal{P}caligraphic_P, and ℛℛ\mathcal{R}caligraphic_R modalities.

🔼 Figure 8 shows the performance of CrossOver on cross-modal scene retrieval using ScanNet. The model retrieves scenes from point cloud data (P) given an image query (I). The plots illustrate different performance metrics: scene matching recall (overall accuracy), category recall (retrieving scenes of the same type), temporal recall (retrieving the same scene across different time points), and intra-category recall (retrieving a specific scene instance from a set of similar scenes). The analysis is conducted for varying numbers of camera views (1, 5, 10, 15, 20) used during training to assess impact on retrieval performance, across multiple top-k results (k=1, 5, 10, 20). The maximum k value varies depending on the metric because the number of eligible matches differs based on the criteria for each metric (scene similarity, category, temporal changes).

read the caption

Figure 8: Cross-Modal ℐ→𝒫→ℐ𝒫\mathcal{I}\rightarrow\mathcal{P}caligraphic_I → caligraphic_P Scene Retrieval on ScanNet. Plots showcase scene matching recall (Recall), category recall, temporal recall, and intra-category recall for different number of camera views evaluated on several Top-k𝑘kitalic_k matches. Note that maximum k𝑘kitalic_k differs per recall since the amount of eligible matches depends on the criteria for each recall type: scene similarity, category, temporal changes.

🔼 Figure 9 shows the results of cross-modal scene retrieval experiments on the 3RScan dataset. Specifically, it examines how well the model can retrieve a scene represented by point clouds (𝒫) given a query scene represented by images (ℐ). The experiments vary the number of camera views used to generate the image representation of the query scene. The plots in the figure show both the scene matching recall and the temporal recall for different numbers of camera views. Scene matching recall measures the accuracy of retrieving the correct scene, while temporal recall measures the accuracy of retrieving the correct scene across different time points (i.e., capturing the same scene at different times).

read the caption

Figure 9: Cross-Modal ℐ→𝒫→ℐ𝒫\mathcal{I}\rightarrow\mathcal{P}caligraphic_I → caligraphic_P Scene Retrieval on 3RScan. Plots showcase scene matching recall (Recall) and temporal recall for different number of camera views.

🔼 Table 2 presents the results of cross-modal scene retrieval experiments conducted on the 3RScan dataset. It shows the performance of the CrossOver model in retrieving scenes across different modalities (image, point cloud, text description). The table likely compares CrossOver’s performance to other baselines or previous methods. The caption indicates that the results are similar to those obtained on the ScanNet dataset (reported in Table 3), suggesting consistent performance across different datasets.

read the caption

Table 2: Cross-Modal Scene Retrieval on 3RScan. Similar performance to the ScanNet results in Tab. 3 is observed.

🔼 Table 3 presents the results of cross-modal scene retrieval experiments conducted on the ScanNet dataset. The table compares the performance of the proposed CrossOver model against several state-of-the-art baselines across various metrics, including scene matching recall at different thresholds (R@25%, R@50%, R@75%), scene category recall, temporal recall, and intra-category recall. The results demonstrate that CrossOver consistently outperforms the baselines in most scenarios, showcasing its ability to effectively retrieve scenes across different modalities. However, the self-baseline (a simpler version of the CrossOver model) shows slightly better performance in some intra-category retrieval tasks, particularly for the image to text modality pair (I→R). This difference is attributed to the limitations of the text encoder used in CrossOver, suggesting an area for future improvement.

read the caption

Table 3: Cross-Modal Scene Retrieval on ScanNet. We consistently outperform state-of-the-art methods and our self-baseline in most cases. The latter performs better in certain modality pairs on intra-category, with the biggest gap observed in ℐ→ℛ→ℐℛ\mathcal{I}\rightarrow\mathcal{R}caligraphic_I → caligraphic_R; this can be attributed to our less powerful text encoder.

🔼 This ablation study investigates the robustness of CrossOver’s instance matching performance when training data for different modalities are sourced from separate repositories, resulting in non-overlapping data. The table shows that even with significantly less data in one or both modality pairs, the model maintains high performance, demonstrating its ability to generalize and leverage relationships between modalities even when training data is limited or incomplete.

read the caption

Table 4: Ablation on 𝒫→ℳ→𝒫ℳ\mathcal{P}\rightarrow\mathcal{M}caligraphic_P → caligraphic_M instance matching on ScanNet with non-overlapping data per modality pair. Despite modality pairs not sharing the same image repository, our method retains high performance even when a pair is underrepresented in the data.

🔼 This table presents the results of scene-level matching on the ScanNet dataset. Two sets of results are shown: one where the instance-level encoder was trained using all pairwise combinations of modalities (‘All Pairs’) and one using the method described in the paper (‘Ours’). The results show that training with all pairwise modalities does not lead to a significant improvement in performance compared to the proposed method. This is true even for modality pairs that are not directly aligned in the ‘Ours’ (emergent) results. This suggests that the proposed method’s approach of focused modality alignment is more effective.

read the caption

Table B.1: Scene-level matching results on ScanNet. ‘All Pairs’ refers to training our instance-level encoder with all pairwise modality combinations. As shown, training on all pairwise combinations does not provide drastically improved performance, as one would expect, even in the modality pairs that are not directly aligned in ‘Ours’ (emergent).

🔼 This table presents a comparison of scene-level matching recall results on the 3RScan dataset between two training methods: one using all pairwise modality combinations during instance-level encoder training, and the other (‘Ours’) using a more selective approach. The results are reported for various combinations of modalities (I-P, I-R, P-R), showing recall rates at 25%, 50%, and 75% thresholds. The comparison highlights that training with all pairwise combinations does not consistently improve performance compared to the selective approach, even for modality pairs where direct alignment wasn’t explicitly performed during training. The ’emergent’ performance of the selective method suggests a capability to learn cross-modal relationships implicitly.

read the caption

Table B.2: Scene-level matching results on 3RScan. ‘All Pairs’ refers to training our instance-level encoder with all pairwise modality combinations. Similar to ScanNet, training on all pairwise combinations does not provide improved performance, as one would expect, even in the modality pairs that are not directly aligned in ‘Ours’ (emergent).

🔼 Table C.1 presents the results of same-modality scene retrieval experiments conducted on the ScanNet dataset. The table compares the performance of the proposed CrossOver method against several baseline methods (ULIP-2, PointBind, and an instance-level baseline) across various metrics, including top-k scene matching recall, scene category recall, temporal recall, and intra-category recall. The results demonstrate that CrossOver achieves comparable or superior performance to the baselines across all metrics. This finding supports the claim that the individual modalities within CrossOver’s learned embedding space are well-aligned, even though the model was primarily trained with a cross-modal objective.

read the caption

Table C.1: Same-Modality Scene Retrieval on ScanNet. Our method performs on par with or better than baselines in same-modality scene retrieval across most metrics, indicating that individual modalities in our method are closely aligned within the embedding space, despite the cross-modal training objective.

🔼 Table C.2 presents the results of same-modality scene retrieval experiments conducted on the 3RScan dataset. It compares the performance of the proposed CrossOver method against several established baselines (ULIP-2, PointBind, and an instance-level baseline) across various metrics including top-k scene retrieval recall (top-1, top-5, top-10), scene category recall, temporal recall, and intra-category recall. The results show that CrossOver achieves comparable or better performance than the baselines, despite the limited training data available for the 3RScan dataset, which likely accounts for the slightly lower overall performance compared to the ScanNet results presented in Table C.1.

read the caption

Table C.2: Same-Modality Scene Retrieval on 3RScan. Our method performs on par with or better than baselines in same-modality scene retrieval across most metrics. The lower performance on this dataset is likely due to limited training data availability.

🔼 This table compares the performance of three different approaches for cross-modal scene retrieval on the ScanNet dataset. The methods compared are: 1) a unified encoder trained using all pairwise modality combinations, 2) a scene-level encoder using a single modality input, and 3) the proposed CrossOver model. The table shows that the CrossOver model generally outperforms both the all-pairs method and the uni-modal scene-level approach in terms of scene matching recall, scene category recall, temporal recall, and intra-category recall. A key finding is that the scene-level encoder’s performance is significantly affected by its dependence on instance-level data, whereas the CrossOver model’s unified dimensionality encoders are more robust to the absence of specific modalities.

read the caption

Table D.1: Uni-modal & All pair-wise modality training on Scene-Level Encoder Inference on Cross-Modal Scene Retrieval on ScanNet. ‘All Pairs’ refers to training our unified encoder with all pairwise modality combinations. ‘Uni-modal’ refers to the scene-level encoder with single-modality input. As shown in the Table, our approach outperforms the scene-level encoder and ‘All Pairs’ in most cases. Unlike the unified dimensionality encoders, the scene-level encoder relies on instance-level data, even when operating on a single modality.

🔼 This table presents a comparison of cross-modal scene retrieval performance on the 3RScan dataset using different training methods for the scene-level encoder. The methods compared are: using all pairwise modality combinations during training (‘All Pairs’), using only a single input modality (‘Uni-modal’), and the proposed CrossOver method (‘Ours’). The metrics used are scene matching recall (top-1, top-5, top-10, top-20) and temporal recall (top-1, top-5, top-10). The results show that CrossOver significantly outperforms the uni-modal scene-level encoder in most cases and generally performs better than the ‘All Pairs’ training approach. The superior performance of CrossOver is attributed to its use of unified dimensionality encoders that do not rely on instance-level data, unlike the scene-level encoder.

read the caption

Table D.2: Uni-modal & All pair-wise modality training on Scene-Level Encoder Inference on Cross-Modal Scene Retrieval on 3RScan. ‘All Pairs’ refers to training our unified encoder with all pairwise modality combinations. ‘Uni-modal’ refers to the scene-level encoder with single-modality input. As shown in the Table, our approach outperforms the scene-level encoder in all but one case. Unlike the unified dimensionality encoders, the scene-level encoder relies on instance-level data, even when operating with a single modality.

🔼 Table E.1 presents the results of cross-modal coarse visual localization experiments conducted on the 3RScan dataset. The task is to retrieve a scene’s corresponding multi-modal map given a single image of that scene. The evaluation uses the experimental setup from SceneGraphLoc [29]. The table compares the performance of the proposed method to SceneGraphLoc and its baselines (LidarCLIP [19] and LipLoc [36]). Despite utilizing less information within its multi-modal maps, the proposed method demonstrates competitive performance with SceneGraphLoc.

read the caption

Table E.1: Cross-Modal Coarse Visual Localization on 3RScan. Given a single image of a scene, the goal is to retrieve the corresponding scene from a database of multi-modal maps. We evaluate following the experimental setup in SceneGraphLoc [29] and compare our method to it and its baselines. Despite encoding less information in our multi-modal maps, our method performs competitively with SceneGraphLoc.