Feature4X: Bridging Any Monocular Video to 4D Agentic AI with Versatile Gaussian Feature Fields

🔼 Figure 2 illustrates the Feature4X framework’s architecture for processing monocular video input into a 4D representation suitable for various AI tasks. 2D priors are used to delineate static and dynamic elements. Static elements are represented by static 3D Gaussians combined with latent features, while dynamic elements are modeled using dynamic 3D Gaussians guided by Motion Scaffolds (a graph structure encoding motion trajectories and latent features). These Gaussians are then combined and processed through a parallel rasterization technique generating RGB images and a unified latent feature map. This map is subsequently decoded into task-specific features, as exemplified using SAM2, CLIP-LSeg, and InternVideo2 for 2D, 3D, and 4D tasks respectively. The framework’s adaptability to various 2D vision models and end-to-end training approach are also highlighted, demonstrating its ability to seamlessly support user prompts and LLM interactions for creating a unified 4D agentic AI system.

read the caption

Figure 2: Method overview. Given an input monocular video, we infer 2D priors to segment static background (represented by static 3D Gaussians augmented with latent features) and dynamic foreground (represented by dynamic 3D Gaussians guided by Motion Scaffolds [35], a set of nodes {𝐯i}subscript𝐯𝑖\{\mathbf{v}_{i}\}{ bold_v start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT } encoding 3D motion trajectories and latent features hisubscriptℎ𝑖h_{i}italic_h start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT). Dynamic Gaussian features and motions are computed via interpolation from their K𝐾Kitalic_K-nearest scaffold nodes. At each timestep, dynamic Gaussians are warped and fused with static Gaussians. A parallel rasterization [102] generates RGB images and a unified latent feature map, decoded into task-specific features—illustrated here by SAM2 [68], CLIP-LSeg [36], and InternVideo2 [84] for representative 2D (novel view segmentation), 3D (scene editing), and 4D (spatiotemporal VQA) tasks. Our framework generalizes to any 2D vision foundation model and is trained end-to-end using input RGB frames and customized features from pretrained 2D models. At inference, rendered feature maps from arbitrary views and timesteps are directly fed into task-specific decoders, seamlessly supporting user prompts and LLM interactions to form a unified 4D agentic AI system.

🔼 Figure 3 demonstrates the capabilities of Feature4X in performing semantic segmentation on dynamic 4D scenes using the Segment Anything Model (SAM2). It showcases two modes: (a) Promptless segmentation, where the system automatically segments all objects in the first frame (t=0) and propagates these masks consistently across subsequent frames, handling both static and dynamic content. (b) Promptable segmentation, which allows users to interactively segment any object (static or dynamic) at any point in time simply by providing a point or box prompt. The system then accurately tracks and propagates this mask through the rest of the video. This highlights the versatility and robustness of Feature4X in handling complex 4D scenes.

read the caption

Figure 3: Segment Anything in Dynamic 4D Scenes with SAM2 Feature Field. For any rendered novel view video, we support: (a) Promptless segmentation (segment everything): when no user prompt is provided, segmentation masks are automatically assigned at the first frame (t=0𝑡0t=0italic_t = 0) and then propagated across all frames. (b) Promptable segmentation (segment anything): the user can segment any object—static or dynamic—at any timestep using a point or box prompt, and the corresponding mask is robustly tracked and propagated through subsequent frames.

🔼 Figure 4 presents a comparison of two methods for SAM2 inference: a naive RGB-based approach and a novel feature-based approach. The RGB-based approach processes the raw RGB video frames directly through the SAM2 model, while the feature-based approach utilizes a learned compact 4D feature field representation to achieve segmentation. The results demonstrate that the feature-based method achieves comparable segmentation accuracy, accurately tracking the object across frames. Importantly, it avoids the RGB artifacts observed in the naive RGB-based approach, specifically around time step 70. The feature-based method also offers a significant speed-up in inference time, achieving approximately four times faster processing than the RGB-based approach.

read the caption

Figure 4: Baseline Comparison on SAM2 Inference. We compare segmentation quality and inference speed between (a) the naive RGB-based approach and (b) our feature-based method. Ours achieves comparable segmentation, accurately tracking the object over time, and avoids RGB artifacts (red box region at t=70𝑡70t=70italic_t = 70), while reducing inference time to about 4×\times× speed-up.

🔼 Figure 5 demonstrates the capability of Feature4X to perform semantic 4D scene understanding. By incorporating CLIP-LSeg features into a 4D feature field, the model achieves pixel-level semantic segmentation from any viewpoint at any point in time. This addresses a key challenge in 3D/4D vision: maintaining consistent semantic understanding even as object appearance changes dynamically throughout the video. The example of a blooming flower, accurately identified from bud to full bloom across multiple views, showcases this robust 4D scene understanding capability.

read the caption

Figure 5: Semantic 4D Scene Understanding with CLIP Feature Field. By lifting CLIP-LSeg [36] features into a 4D feature field, we enable pixel-level semantic segmentation from any view at any timestep. This allows robust 4D scene understanding, even as object appearances change over time—for example, accurately identifying a blooming flower from bud to full bloom across views.

🔼 This figure showcases the capabilities of the LLM-powered AI agent in editing 4D scenes. The agent takes high-level natural language instructions (e.g., ‘Extract the swan’, ‘Delete the moving camel’, ‘Change the cow color to black and white’) and uses the 4D CLIP feature field to perform the edits. The edits can involve geometric changes (like extracting or deleting objects) or changes in appearance (like altering an object’s color). While the results might not be perfect due to limitations in aligning fine-grained features or finding optimal editing parameters, the AI agent iteratively refines these parameters and ensures consistent application of edits across different viewpoints and time steps. This significantly simplifies the editing process and enables robust, interactive 4D scene manipulation.

read the caption

Figure 6: Scene Editing with AI Agent. Given user prompts, our GPT-powered agent interprets editing intent and autonomously performs scene edits via our 4D CLIP feature field. Examples include both geometric (e.g., “extract” and “delete”) and appearance (e.g., “change color”) editing in 3D space. While results may not be perfect due to imperfect fine-grained feature alignment and non-optimal editing parameter tuning, the agent adaptively refines parameters and applies edits consistently across views and time—greatly reducing the need for manual tuning—and demonstrates robust, interactive 4D scene manipulation.

🔼 Figure 7 illustrates the model’s ability to answer visual questions (VQA) using a chatbot interface. The left side shows the diverse question types supported: general, spatial, and temporal questions. The core of this capability comes from leveraging InternVideo2’s features. The right panel highlights the model’s 4D reconstruction, producing both a 4D radiance field and a 4D feature field. This provides access to richer inference sources than just the original input video. These include novel views (both local, from a moving camera, and global, zoomed-out) and their corresponding feature maps. This approach is crucial for robust spatiotemporal reasoning in VQA.

read the caption

Figure 7: VQA with Chatbot Agent. (Left) Our model supports free-form VQA across diverse question types—general, spatial, and temporal—by distilling InternVideo2 [84] features. (Right) At each timestep, we reconstruct both a 4D radiance field and a 4D feature field, providing more inference sources beyond the input video frame—including local (moving camera) and global (zoomed-out) novel views and their corresponding feature maps—thereby supporting VQA in 4D and enhancing the model’s spatiotemporal reasoning capabilities.

🔼 This figure visualizes the versatile Gaussian feature field developed in the Feature4X framework. It uses Principal Component Analysis (PCA) to reduce the dimensionality of the feature fields for better visualization. The visualization includes the unified latent feature field and its decoded versions for three different 2D foundation models: Segment Anything (SAM2), CLIP-LSeg, and InternVideo. Each model’s decoded feature field provides a different perspective on the underlying 4D scene representation, highlighting the versatility of the Feature4X framework in adapting to various downstream tasks.

read the caption

Figure A: Feature Field Visualizations. We visualize our versatile Gaussian feature field along with its decoded SAM2, CLIP, and InternVideo feature fields using PCA.

🔼 This figure illustrates the 4D scene editing framework. It begins with a user providing a high-level prompt (e.g., ‘Make the dog’s color look like Clifford’). This prompt is processed by GPT-40, a large language model, to generate different editing configurations. These configurations include details like which objects to edit, what operations to perform, the specific targets for the edits, and threshold settings. The model then uses a hybrid filtering method to identify the relevant Gaussians (points in a 3D Gaussian splatting representation) within the 4D scene that correspond to the targeted objects. The system then applies the selected edit operation and renders the result. Finally, the GPT-40 model evaluates the rendered image quality and selects the best configuration, ensuring consistency and high quality results across the 4D scene.

read the caption

Figure B: Overview of the editing framework. GPT-4o generates different editing configurations based on user prompts, selects target regions via hybrid filtering, evaluates their outputs, and selects the best configuration.

🔼 This figure compares the semantic segmentation results obtained using CLIP features. Two methods are compared: one using ground-truth RGB images and the other using the proposed method’s feature maps. The comparison is done for both training and novel (unseen) views to show how well the proposed method generalizes to new viewpoints. The results visually demonstrate the effectiveness of the proposed method for accurate and consistent semantic segmentation across different views, even without directly using the full RGB image information.

read the caption

Figure C: CLIP semantic segmentation quality comparison. We compare the CLIP semantic segmentation quality between ground-truth (inference from RGB) and our implementation (inference from feature map) for both training and novel views.

🔼 This figure compares the segmentation quality achieved using the Segment Anything Model 2 (SAM2) with different dimensions for the unified latent feature maps. The results show that a 32-dimensional unified latent feature map produces the best performing SAM2 segmentation, accurately tracking the object over time and avoiding artifacts present in the results from other dimensions.

read the caption

Figure D: SAM2 segmentation quality comparison for different dimensions of unified latent feature maps Best performing SAM2 segmentation is derived from the 32-dimensional unified latent feature map.

🔼 This figure shows the training time taken for different dimensions of the unified latent feature map. It demonstrates the trade-off between model complexity and training efficiency. As the dimension of the latent feature increases, so does the training time. This is because higher dimensions increase the number of parameters the model needs to learn, leading to longer training times.

read the caption

Figure E: Training Time vs Unified Latent Feature Dimensions We show the training time required with different dimensions of unified latent feature map.

🔼 This figure demonstrates the relationship between the dimensionality of the unified latent feature map and the time needed for rendering. As the dimensionality increases, the rendering time also increases, highlighting a trade-off between feature representation complexity and computational efficiency. The plot likely shows an exponential or near-exponential increase in rendering time as the dimension increases, indicating that higher-dimensional features significantly increase computational demands during the rendering process.

read the caption

Figure F: Rendering Time vs Unified Latent Feature Dimensions We show the rendering time required for different dimensions of unified latent feature map.

🔼 This figure shows a graph illustrating the relationship between the training time and the dimensionality of rendered CLIP features. Higher dimensional rendered CLIP features lead to longer training times. This demonstrates a trade-off between model complexity and training efficiency.

read the caption

Figure G: Training Time vs CLIP Feature Dimensions We show the training time required with different dimensions of rendered CLIP features.

🔼 Figure H illustrates the relationship between the rendering time and the dimensions of rendered CLIP features. The x-axis represents the different dimensions of the rendered CLIP features used in the experiment, while the y-axis shows the corresponding rendering time. The graph likely demonstrates that increasing the dimensions of the CLIP features significantly increases the rendering time required. This suggests a trade-off between feature richness and computational efficiency.

read the caption

Figure H: Rendering Time vs CLIP Feature Dimensions We show the rendering time required for different dimensions of rendered CLIP features.

🔼 This figure presents the results of an ablation study that examines how different dimensions of rendered CLIP features affect the mean Intersection over Union (mIoU) score in semantic segmentation. The x-axis represents the dimension of the rendered CLIP features, while the y-axis shows the corresponding mIoU score. The graph helps to determine the optimal dimensionality for achieving a balance between performance and computational efficiency.

read the caption

Figure I: mIoU vs CLIP Feature Dimensions We show mIoU with respect to different rendered CLIP feature dimensions.

🔼 This table presents the performance of Visual Question Answering (VQA) using different inference sources on the scene depicted in Figure 7. The inference sources are categorized into view-based (input video view, local novel view, global novel view) and feature-based (local novel view feature, global novel view feature). The table shows the accuracy of answering general, spatial, and temporal questions using each inference source. It highlights that feature-based inference achieves higher accuracy and requires less time for all question types compared to view-based methods, indicating that feature-based inference is better suited for spatiotemporal reasoning tasks.

read the caption

Table 2: VQA performance across different inference sources with the scene shown in Fig. 7. Feature-based inference supports all question types with lower latency, while view-based methods are limited in spatial and temporal reasoning.

🔼 This table presents a quantitative comparison of spatiotemporal Visual Question Answering (VQA) performance on the DAVIS dataset. It compares the accuracy and inference time of different methods for answering various types of questions (general, spatial, and temporal) about a video scene. The methods include using only the input video, local novel views, features from local novel views, global novel views, and features from global novel views. The key finding is that using our 4D feature space (Global Novel Feature) significantly improves the Video-LLM’s ability to reason about both space and time in the video, while also achieving a threefold speedup in inference compared to 2D video inference.

read the caption

Table 3: Spatiotemporal VQA on DAVIS dataset [61]. Compared to 2D video inference, our 4D feature space inference (Global Novel Feature) enhances the Video-LLM’s spatiotemporal reasoning while achieving approximately 3×\times× faster inference.

🔼 This table presents the mean Intersection over Union (mIoU) scores achieved by the Segment Anything Model 2 (SAM2) on the NVIDIA and Nerfies datasets. It compares the performance of two approaches: a baseline method using RGB images directly and the proposed Feature4X method which uses distilled features. The results are presented for three separate experiments (Exp1, Exp2, Exp3) to showcase the robustness and stability of the Feature4X method. Inference time is also reported for each approach.

read the caption

Table A: SAM2 Quantitative Results (mIoU) on NVIDIA and Nerfies Datasets.

🔼 Table B presents a detailed quantitative analysis of the proposed Feature4X model’s performance on seven distinct scenes from the NVIDIA dataset. For each scene, the table compares the performance metrics of four different methods: the baseline MoSca model, MoSca augmented with Feature 3DGS, Feature4X with a single CLIP head, and the full Feature4X model. The metrics reported include PSNR, SSIM, LPIPS, accuracy, and mIoU, providing a comprehensive evaluation of the model’s ability to reconstruct the scene accurately, and its effectiveness in performing both semantic segmentation and spatiotemporal reasoning. The table also shows the model sizes (static and dynamic) for each method, offering insights into memory usage and computational efficiency.

read the caption

Table B: Detailed Performance of 7 scenes from the NVIDIA Dataset.

🔼 This table presents a quantitative comparison of semantic segmentation performance on the Nerfies dataset. It compares the mean Intersection over Union (mIoU), accuracy, and inference time for different methods, including RGB-based and feature-based approaches, across several scenes (Broom, Curls, Tails, Toby-sit). The results showcase the improved efficiency and comparable segmentation accuracy achieved by the feature-based method compared to the RGB-based method.

read the caption

Table C: Semantic Segmentation Quantitative Results on Nerfies Dataset.

🔼 This table presents a comprehensive analysis of semantic segmentation performance on the NVIDIA Jumping Scene dataset, using different dimensions for the unified latent feature field. It evaluates the impact of varying the feature dimension on training time (in hours), rendering time (in seconds), mean Intersection over Union (mIoU), and accuracy. Higher dimensions offer potentially more detailed representations, but this comes at a cost of increased computational expense.

read the caption

Table D: Evaluation of Semantic Segmentation Performance On NVIDIA Jumping Scene Across Different Dimensions. This table presents the Time, mIoU, and Accuracy corresponding to each dimension level.

🔼 This table presents a quantitative analysis of image quality across various dimensions of the unified latent feature field used in the Feature4X model. Specifically, it shows how Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) metrics change as the dimensionality of the feature field is altered. This helps assess the impact of feature dimensionality on the visual fidelity of the reconstructed scenes, specifically for the ‘Jumping’ scene from the NVIDIA dataset.

read the caption

Table E: Evaluation of Image Quality Metrics On NVIDIA Jumping Scene Across Different Dimensions. This table presents the PSNR, SSIM, and LPIPS values corresponding to each dimension level.