VGGT: Visual Geometry Grounded Transformer

🔼 This figure illustrates the architecture of the Visual Geometry Grounded Transformer (VGGT) model. The process begins with input images that are initially divided into patches using the DINO (DETR with Improved DeNoising) method. These image patches are then converted into tokens, which are numerical representations used by the transformer network. To predict camera parameters, the model adds special ‘camera tokens’ to the image tokens. The core of VGGT is a transformer network that uses alternating layers of frame-wise and global self-attention. Frame-wise attention processes information within each individual image, while global self-attention integrates information across all input images. Finally, a dedicated ‘camera head’ in the network produces the final estimates for camera extrinsics (position and orientation) and intrinsics (focal length and sensor dimensions). A separate DPT (Depth Prediction Transformer) head generates the dense outputs such as depth maps and point clouds.

read the caption

Figure 2: Architecture Overview. Our model first patchifies the input images into tokens by DINO, and appends camera tokens for camera prediction. It then alternates between frame-wise and global self attention layers. A camera head makes the final prediction for camera extrinsics and intrinsics, and a DPT [87] head for any dense output.

🔼 This figure qualitatively compares the 3D point cloud predictions of the proposed VGGT method and the DUSt3R method on real-world images. It showcases VGGT’s superior performance in three scenarios: reconstructing the geometry of an oil painting (where DUSt3R produces a distorted plane), recovering a 3D scene from only two images with no overlap (a task DUSt3R fails), and handling an image with repetitive textures (where VGGT maintains high quality). The figure highlights VGGT’s ability to produce accurate 3D reconstructions even in challenging situations, unlike DUSt3R, which is limited by memory constraints for scenes with more than 32 images.

read the caption

Figure 3: Qualitative comparison of our predicted 3D points to DUSt3R on in-the-wild images. As shown in the top row, our method successfully predicts the geometric structure of an oil painting, while DUSt3R predicts a slightly distorted plane. In the second row, our method correctly recovers a 3D scene from two images with no overlap, while DUSt3R fails. The third row provides a challenging example with repeated textures, while our prediction is still high-quality. We do not include examples with more than 32 frames, as DUSt3R runs out of memory beyond this limit.

🔼 Figure 4 presents supplementary visualizations enhancing the understanding of point map estimation from the paper. It showcases multiple examples of 3D point cloud reconstructions, with camera frustums overlaid to clearly illustrate the estimated camera poses and orientations for each scene. The caption encourages users to explore an interactive demo for an improved and more detailed visualization experience, suggesting that the static figure is a limited representation of the full capabilities of the model.

read the caption

Figure 4: Additional Visualizations of Point Map Estimation. Camera frustums illustrate the estimated camera poses. Explore our interactive demo for better visualization quality.

🔼 Figure 5 demonstrates the application of VGGT’s tracking module for both static and dynamic scenes. The top row showcases the tracking module’s ability to generate keypoint tracks from an unordered set of images depicting a static scene. The bottom row illustrates how finetuning VGGT’s backbone improves the performance of a dynamic point tracker (CoTracker), which typically processes sequential inputs. This highlights VGGT’s versatility and adaptability to various tracking scenarios.

read the caption

Figure 5: Visualization of Rigid and Dynamic Point Tracking. Top: VGGT’s tracking module 𝒯𝒯\mathcal{T}caligraphic_T outputs keypoint tracks for an unordered set of input images depicting a static scene. Bottom: We finetune the backbone of VGGT to enhance a dynamic point tracker CoTracker [56], which processes sequential inputs.

🔼 This figure showcases qualitative examples of novel view synthesis. The top row displays the input images used for synthesis. The middle row shows the corresponding ground truth images from the target viewpoints, providing a reference for comparison. The bottom row presents the novel views synthesized by the VGGT model, illustrating its ability to generate realistic and visually consistent images from unseen perspectives.

read the caption

Figure 6: Qualitative Examples of Novel View Synthesis. The top row shows the input images, the middle row displays the ground truth images from target viewpoints, and the bottom row presents our synthesized images.

🔼 This table presents a comparison of different methods for dense multi-view stereo (MVS) estimation on the DTU dataset. The methods are categorized into two groups: those that utilize ground-truth camera poses (top section) and those that do not (bottom section). The performance is evaluated using Accuracy, Completeness, and Overall metrics (average of Accuracy and Completeness). This demonstrates the capabilities of methods operating under ideal conditions versus those working with real-world constraints where ground truth camera information might not be available.

read the caption

Table 2: Dense MVS Estimation on the DTU [51] Dataset. Methods operating with known ground-truth camera are in the top part of the table, while the bottom part contains the methods that do not know the ground-truth camera.

🔼 This table presents a comparison of point map estimation results on the ETH3D dataset among different methods. It highlights the speed advantage of the proposed feed-forward approach (VGGT) compared to optimization-based methods (DUSt3R and MASt3R) which rely on global alignment. The table also shows that combining depth and camera predictions to construct the point cloud (Ours (Depth + Cam)) results in better accuracy than directly using the point map head output (Ours (Point)). Metrics used include Accuracy, Completeness, and Overall (Chamfer distance).

read the caption

Table 3: Point Map Estimation on ETH3D [97]. DUSt3R and MASt3R use global alignment while ours is feed-forward and, hence, much faster. The row Ours (Point) indicates the results using the point map head directly, while Ours (Depth + Cam) denotes constructing point clouds from the depth map head combined with the camera head.

🔼 Table 4 presents a comparison of different methods for two-view image matching on the ScanNet-1500 dataset. The table shows the performance of various techniques, including the proposed VGGT method and state-of-the-art approaches, in terms of AUC (Area Under the Curve) scores, a common metric for evaluating matching performance. The results demonstrate that despite not being specifically designed for two-view matching, VGGT outperforms the existing best-performing method, Roma, indicating its strong generalizability across different 3D vision tasks.

read the caption

Table 4: Two-View matching comparison on ScanNet-1500 [18, 92]. Although our tracking head is not specialized for the two-view setting, it outperforms the state-of-the-art two-view matching method Roma. Measured in AUC (higher is better).

🔼 This ablation study investigates the impact of different attention mechanisms on the performance of the VGGT model for point map estimation using the ETH3D dataset. Three transformer backbone architectures are compared: one with only global self-attention, one with cross-attention, and the proposed alternating-attention mechanism. The results show the effectiveness of the alternating-attention approach in improving accuracy compared to the other two methods.

read the caption

Table 5: Ablation Study for Transformer Backbone on ETH3D. We compare our alternating-attention architecture against two variants: one using only global self-attention and another employing cross-attention.

🔼 This table presents an ablation study evaluating the impact of multi-task learning on point map estimation accuracy within the ETH3D dataset. By selectively including or excluding the training of camera, depth, and track estimation components, the study demonstrates that a multi-task learning approach, encompassing all three elements, results in superior point map estimation accuracy compared to single-task learning.

read the caption

Table 6: Ablation Study for Multi-task Learning, which shows that simultaneous training with camera, depth and track estimation yields the highest accuracy in point map estimation on ETH3D.

🔼 This table presents a quantitative comparison of different methods for novel view synthesis using the GSO dataset. The key metric is the performance of each method in generating new views of a scene, measured using PSNR, SSIM, and LPIPS scores. The table highlights the competitive performance of VGGT even when trained on a limited dataset (only 20% of the available data) and without requiring explicit knowledge of camera parameters.

read the caption

Table 7: Quantitative comparisons for view synthesis on GSO [28] dataset. Finetuning VGGT for feed-forward novel view synthesis, it demonstrates competitive performance even without knowing camera extrinsic and intrinsic parameters for the input images. Note that ∗ indicates using a small training set (only 20%).

🔼 This table presents the results of dynamic point tracking experiments using the TAP-Vid benchmark. The authors compared their method (CoTracker fine-tuned with weights from their pre-trained Visual Geometry Grounded Transformer (VGGT) model) against other state-of-the-art tracking methods. The results demonstrate that even though the VGGT model wasn’t specifically trained for dynamic scenes, integrating its learned features significantly improves the performance of CoTracker, highlighting the robustness and generalizability of VGGT’s feature extraction capabilities. Metrics such as Occlusion Accuracy (OA), Average Jaccard (AJ), and others are used to evaluate the tracker’s performance across various datasets. The findings strongly support the effectiveness of VGGT’s learned representations when applied to downstream tasks.

read the caption

Table 8: Dynamic Point Tracking Results on the TAP-Vid benchmarks. Although our model was not designed for dynamic scenes, simply fine-tuning CoTracker with our pretrained weights significantly enhances performance, demonstrating the robustness and effectiveness of our learned features.

🔼 This table presents the runtime and GPU memory usage of the VGGT model’s feature backbone when processing varying numbers of input images. The runtime is measured in seconds, and the peak GPU memory usage is reported in gigabytes. This data allows readers to assess the computational efficiency and resource requirements of the model as the number of input frames increases.

read the caption

Table 9: Runtime and peak GPU memory usage across different numbers of input frames. Runtime is measured in seconds, and GPU memory usage is reported in gigabytes.

🔼 Table 10 presents a comparison of camera pose estimation methods on the Image Matching Challenge (IMC) dataset [54], focusing on phototourism data. The results show that the proposed method achieves state-of-the-art performance, surpassing even VGGSfMv2 [125], the top performer in the recent CVPR 2024 IMC challenge. The table evaluates the accuracy of camera pose estimation (both rotation and translation) using various metrics and compares the performance of different algorithms, highlighting the speed and accuracy advantages of the proposed approach.

read the caption

Table 10: Camera Pose Estimation on IMC [54]. Our method achieves state-of-the-art performance on the challenging phototropism data, outperforming VGGSfMv2 [125] which ranked first on the latest CVPR’24 IMC Challenge in camera pose (rotation and translation) estimation.