Wonderland: Navigating 3D Scenes from a Single Image

🔼 Given a single image, Wonderland reconstructs 3D scenes using a camera-guided video diffusion model and a latent-based large reconstruction model (LaLRM). The video diffusion model, conditioned on the input image and camera trajectory, generates a 3D-aware video latent. This latent is then used by the LaLRM to construct the 3D scene in a feed-forward manner. The dual-branch camera conditioning in the video diffusion model ensures precise pose control, while the LaLRM’s operation in latent space allows for efficient reconstruction of wide-scope, high-fidelity 3D scenes.

read the caption

Figure 2: Overview of Wonderland. Given a single image, a camera-guided video diffusion model follows the camera trajectory and generates a 3D-aware video latent, which is leveraged by the latent-based large reconstruction model to construct the 3D scene in a feed-forward manner. The video diffusion model involves dual-branch camera conditioning to fulfill precise pose control. The LaLRM operates in latent space and efficiently reconstructs a wide-scope and high-fidelity 3D scene.

🔼 This figure provides a qualitative comparison of several camera-guided video generation models, including MotionCtrl, VD3D, ViewCrafter, and the proposed method, Wonderland. Each model is tasked with generating a video sequence conditioned on a single input image and a specified camera trajectory. The 14th frame of each generated sequence is displayed for comparison, along with the ground truth frame and the conditional input image. The comparison highlights Wonderland’s superior performance in terms of precise camera control, high visual quality, and wide-scope scene generation, particularly when compared to other methods that might exhibit misalignment with the specified camera trajectory, limited visual quality, or a narrow scope of generation.

read the caption

Figure 3: Qualitative comparison to prior arts in camera-guided video generation. The 14thsubscript14th14_{\mathrm{th}}14 start_POSTSUBSCRIPT roman_th end_POSTSUBSCRIPT frame in each sample is shown for comparison, with the first column displaying the conditional image and camera trajectory (bottom-right). Blue bounding boxes denote reference areas to assist comparison and orange bounding boxes highlight low-quality generations. We also show our last frames in the rightmost column. Our method outperforms the priors in both precise camera control and high-quality and wide-scope video generation.

🔼 This figure presents a qualitative comparison of novel view synthesis from a single image among Wonderland, ViewCrafter, and ZeroNVS. Each row showcases a different scene, with ground truth on the far left, then the novel view generated by each method. The first two scenes are indoors and the last is outdoors. Blue boxes indicate regions visible in the original image. Yellow boxes highlight areas where comparison methods struggle with quality or consistency. Wonderland shows significant improvement in rendering high-quality and detailed novel views, particularly in unseen or partially occluded regions.

read the caption

Figure 4: Qualitative comparison for 3D scene generation. Blue bounding boxes show visible regions from conditional images and yellow bounding boxes show low-quality regions. Our approach generates much higher quality novel views from one conditional image.

🔼 This figure presents a qualitative comparison of our proposed method, Wonderland, with two existing state-of-the-art approaches, ViewCrafter and WonderJourney, for generating 3D scenes from single in-the-wild images. Each row showcases a different scene. The leftmost column displays the original input image, while the remaining columns display novel views generated by each method. Wonderland excels at producing consistent and realistic renderings, even across wide viewpoints and previously unseen regions of the scene. ViewCrafter’s performance is limited to a narrow field of view around the input image, exhibiting quality degradation as the view range expands. WonderJourney, designed for zoom-out effects, generates novel views with a wider field of view but suffers from blurry outputs and noticeable artifacts.

read the caption

Figure 5: Comparison with ViewCrafter (left) and WonderJourney (right) for in-the-wild 3D scene generation from single input images.

🔼 This figure presents a qualitative comparison of 3D scene generation between Wonderland, ZeroNVS, and Cat3D on scenes from the Mip-NeRF dataset. Each scene’s source image is displayed in the leftmost column. For each method, renderings from two viewpoints are shown: the initial viewpoint (corresponding to the input image) in the top row and a novel view at a 120-degree rotation from the initial viewpoint in the bottom row. This comparison aims to demonstrate the quality and consistency of novel view synthesis generated by each method, particularly in handling out-of-view regions and geometric details.

read the caption

Figure 6: Comparison to ZeroNVS and Cat3D with Mip-Nerf dataset on 3D scene generation from single input images. For each scene, the conditional image is shown in the left-most column. We show renderings from two viewpoints, one at the conditional image (starting) view (upper) and another at around 120°-rotation from the starting view(lower).

🔼 This figure shows a comparison of video generations between two models: a source I2V model and a fine-tuned model. The top row displays output from the original source I2V model. The bottom row displays output from the I2V model fine-tuned on static scene datasets using LoRA. The comparison shows that the fine-tuned model produces significantly more static scenes. Specifically, the fine-tuned model generates videos with fewer motion changes compared to the original source model for scenes containing subjects like people and animals.

read the caption

Figure A1: Comparison of video generations between the source model (upper row) and the model fine-tuned on static-scene datasets with LoRA modules (lower row). The results demonstrate that fine-tuning the model on static-scene datasets equipped with LoRA produces significantly more static scenes.

🔼 This figure compares the 3D rendering performance of two latent reconstruction models: one fine-tuned without in-the-wild data and the other with it. Each model is used to render a pair of novel views of four different scenes. The upper row shows results from the model trained only on benchmark datasets, while the lower row shows results from the model further fine-tuned with an in-the-wild dataset. The inclusion of in-the-wild data during fine-tuning leads to a noticeable improvement in the generalization capability of the model, resulting in higher quality 3D renderings with finer details.

read the caption

Figure A2: Comparison of 3D rendering performance between latent reconstruction models fine-tuned without in-the-wild dataset (upper row) and with in-the-wild dataset (lower row). Involving in-the-wild datasets during fine-tuning improves the generalization capability.

🔼 This architecture diagram outlines the Dual-branch Camera-guided Video Diffusion Model. It illustrates the integration of camera guidance using both a LoRA-branch and a ControlNet-branch during the training process. Random noise is introduced to video latents, while the conditional image is incorporated through feature concatenation. For clarity, text tokens, diffusion time embeddings, positional embeddings, and some reshaping operations are omitted. In the underlying diffusion transformer, text tokens are concatenated with visual tokens along the number-of-token dimension. To maintain consistent length, zero-padding is applied to the camera tokens before concatenation or element-wise summation. The SiLU activation function is used by default.

read the caption

Figure A3: Structure of Dual-branch Camera-guided Video Diffusion Model. We show the skeletons of the training pipeline, where random noise is added to the video latents. The conditional image is merged to the noisy latents via feature concatenation. The camera guidance is integrated with LoRA-branch (left) and ControlNet-branch (right). We ignore the text tokens, the diffusion time embeddings, the positional embeddings, and some reshaping operations for simplicity in the figure. In the foundation diffusion transformer, the text tokens are concatenated along number-of-token dimension with visual tokens. Thus we apply zero-padding to camera tokens to guarantee the same length before concatenation or element-wise sum. By default, we use SiLu as our activation function.

🔼 This table presents a quantitative comparison of different methods for 3D scene novel view synthesis from a single image on various benchmark datasets, including RealEstate10K, DL3DV, and Tanks-and-Temples. The metrics used for evaluation include LPIPS, PSNR, and SSIM, which measure the quality and similarity of the rendered views compared to ground truth views. The table shows the performance of ZeroNVS, ViewCrafter, the proposed method, and an ablation study of the proposed method.

read the caption

Table 2: Quantitative comparison on various benchmark datasets for 3D scene novel view synthesis with single view condition.

🔼 This table provides a comprehensive list of notations used throughout the paper, along with their corresponding specifications and explanations. It serves as a quick reference for readers to understand the meaning of symbols and variables used in equations and figures.

read the caption

Table A1: Overview of the notations used in the paper.

🔼 This table evaluates different architecture designs for a camera-guided video generation model, focusing on visual quality (FID and FVD) and camera control precision (rotation and translation errors). The models are trained on the RealEstate10K dataset. The top section of this table replicates data from Table 1 in the main paper, providing a baseline comparison to other state-of-the-art methods.

read the caption

Table A2: Analysis on architecture designs in camera-guided video generation model. We report the performance for visual quality (FID and FVD) and pose control precision (Rerrsubscript𝑅errR_{\text{err}}italic_R start_POSTSUBSCRIPT err end_POSTSUBSCRIPT and Terrsubscript𝑇errT_{\text{err}}italic_T start_POSTSUBSCRIPT err end_POSTSUBSCRIPT) from models trained on RealEstate10K dataset. The first section of the table is adopted from Tab.1 in the main paper.

🔼 This table evaluates the effect of fine-tuning the Latent Large Reconstruction Model (LaLRM) with an in-the-wild dataset on its ability to perform 3D scene novel view synthesis from single-view images. The evaluation uses three benchmark datasets: RealEstate10K, DL3DV, and Tanks-and-Temples. The metrics used are LPIPS, PSNR, and SSIM, which measure the quality and similarity of rendered images compared to ground truth views. Two versions of the model are evaluated: LaLRM- (trained without the in-the-wild dataset), and LaLRM (trained with the in-the-wild dataset), demonstrating the effectiveness of including in-the-wild data during fine-tuning.

read the caption

Table A3: Analysis on involving in-the-wild dataset to fine-tune LaLRM. We report the performance on various benchmark datasets for novel view synthesis of 3D scenes, which are built from single view condition.