GaussianAnything: Interactive Point Cloud Latent Diffusion for 3D Generation

🔼 This figure illustrates the architecture of the 3D Variational Autoencoder (VAE) within the GaussianAnything model. The VAE takes in multiple views (V) of posed RGB-D-N (Red-Green-Blue, Depth, Normal) renderings of a 3D object as input. These views are initially encoded into an unstructured set latent representation. A cross-attention block then projects this set latent onto a 3D manifold, creating a point-cloud structured latent code (z). A 3D-aware Diffusion Transformer (DiT) decodes this point-cloud latent code, producing an initial, coarse Gaussian prediction for the 3D object. To improve rendering quality, this coarse Gaussian prediction undergoes a series of cascaded upsampling operations (D_U^k), generating a dense Gaussian representation suitable for high-resolution rendering. The training objective for this VAE is detailed in Equation 9 of the paper.

read the caption

Figure 2: Pipeline of the 3D VAE of GaussianAnything. In the 3D latent space learning stage, our proposed 3D VAE ℰϕsubscriptℰbold-italic-ϕ\mathcal{E}_{\bm{\phi}}caligraphic_E start_POSTSUBSCRIPT bold_italic_ϕ end_POSTSUBSCRIPT encodes V−limit-from𝑉V-italic_V -views of posed RGB-D(epth)-N(ormal) renderings ℛℛ\mathcal{R}caligraphic_R into a point-cloud structured latent space. This is achieved by first processing the multi-view inputs into the un-structured set latent, which is further projected onto the 3D manifold through a cross attention block, yielding the point-cloud structured latent code 𝐳𝐳{\mathbf{z}}bold_z. The structured 3D latent is further decoded by a 3D-aware DiT transformer, giving the coarse Gaussian prediction. For high-quality rendering, the base Gaussian is further up-sampled by a series of cascaded upsampler 𝒟Uksuperscriptsubscript𝒟𝑈𝑘\mathcal{D}_{U}^{k}caligraphic_D start_POSTSUBSCRIPT italic_U end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_k end_POSTSUPERSCRIPT towards a dense Gaussian for high-resolution rasterization. The 3D VAE training objective is detailed in Eq. (9).

🔼 This figure illustrates the cascaded diffusion process within the GaussianAnything model for 3D generation. The process begins with a point cloud structured 3D Variational Autoencoder (VAE). Conditional inputs, either text or images, are fed into the DiT architecture (using AdaLN-single and QK-Norm for normalization), interacting via cross-attention blocks at different stages. 3D generation proceeds in two stages: (1) a point cloud diffusion model generates the 3D object’s layout (𝐳x,0), and (2) a texture diffusion model generates the corresponding point cloud features (𝐳h,0), given the initial layout. The combined latent code (𝐳0) is then decoded by the pre-trained VAE to produce the final 3D object.

read the caption

Figure 3: Diffusion training of GaussianAnything. Based on the point-cloud structure 3D VAE, we perform cascaded 3D diffusion learning given text (a) and image (b) conditions. We adopt DiT architecture with AdaLN-single (Chen et al., 2023) and QK-Norm (Dehghani et al., 2023; Esser et al., 2021). For both condition modality, we send in the conditional feature with cross attention block, but at different positions. The 3D generation is achieved in two stages (c), where a point cloud diffusion model first generates the 3D layout 𝐳x,0subscript𝐳𝑥0{\mathbf{z}}_{x,0}bold_z start_POSTSUBSCRIPT italic_x , 0 end_POSTSUBSCRIPT, and a texture diffusion model further generates the corresponding point-cloud features 𝐳h,0subscript𝐳ℎ0{\mathbf{z}}_{h,0}bold_z start_POSTSUBSCRIPT italic_h , 0 end_POSTSUBSCRIPT. The generated latent code 𝐳0subscript𝐳0{\mathbf{z}}_{0}bold_z start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT is decoded into the final 3D object with the pre-trained VAE decoder.

🔼 Figure 4 presents a qualitative comparison of different image-to-3D reconstruction methods on the unseen GSO dataset. Each method is given a single input image, and the results are shown as novel-view 3D reconstructions. The figure highlights the consistent, stable performance of the proposed method across various input images, in contrast to feed-forward methods that, while producing sharper textures, sometimes fail to generate complete and accurate 3D models, especially in challenging scenarios (such as the rhino in the second row). The figure visually demonstrates the superior performance of the proposed native 3D diffusion model in terms of overall 3D reconstruction accuracy.

read the caption

Figure 4: Qualitative Comparison of Image-to-3D. We showcase the novel view 3D reconstruction of all methods given a single image from unseen GSO dataset. Our proposed method achieves consistently stable performance across all cases. Note that though feed-forward 3D reconstruction methods achieve sharper texture reconstruction, these method fail to yield intact 3D predictions under challenging cases (e.g., the rhino in row 2). In contrast, our proposed native 3D diffusion model achieve consistently better performance. Better zoom in.

🔼 Figure 5 showcases a qualitative comparison of text-to-3D generation results achieved by GaussianAnything and several baseline methods. The figure presents two views of 3D objects generated from text prompts. The top section provides a direct comparison between GaussianAnything and baseline methods, demonstrating the superior quality of GaussianAnything’s output. The bottom section presents additional examples generated by GaussianAnything, alongside their corresponding geometry maps, further highlighting the model’s ability to generate high-quality 3D shapes with accurate textures and strong alignment between the generated content and the input text prompt.

read the caption

Figure 5: Qualitative Comparison of Text-to-3D. We present text-conditioned 3D objects generated by GaussianAnything, displaying two views of each sample. The top section compares our results with baseline methods, while the bottom shows additional samples from our method along with their geometry maps. Our approach consistently yields better quality in terms of geometry, texture, and text-3D alignment.

🔼 This figure demonstrates the 3D editing capabilities of the GAUSSIANANYTHING model. Two text prompts are used to generate a point cloud representing the 3D object’s structure (z0,x) using a stage-1 diffusion model and its corresponding features (z0,h) using a stage-2 diffusion model. The stage-2 samples maintain consistent 3D structure but offer diverse textures. The point cloud-structured latent space allows for interactive 3D editing. This is shown by modifying the stage-1 point cloud (z0,x) to (z0,x’) and then regenerating the 3D object using the same Gaussian noise, highlighting the disentanglement of geometry and texture.

read the caption

Figure 6: 3D editing. Given two text prompts, we generate the corresponding point cloud 𝐳0,xsubscript𝐳0𝑥{\mathbf{z}}_{0,x}bold_z start_POSTSUBSCRIPT 0 , italic_x end_POSTSUBSCRIPT with stage-1 diffusion model with ϵΘxsuperscriptsubscriptbold-italic-ϵΘ𝑥\bm{\epsilon}_{\Theta}^{x}bold_italic_ϵ start_POSTSUBSCRIPT roman_Θ end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_x end_POSTSUPERSCRIPT, and the corresponding point cloud features 𝐳0,hsubscript𝐳0ℎ{\mathbf{z}}_{0,h}bold_z start_POSTSUBSCRIPT 0 , italic_h end_POSTSUBSCRIPT can be further generated with ϵΘhsuperscriptsubscriptbold-italic-ϵΘℎ\bm{\epsilon}_{\Theta}^{h}bold_italic_ϵ start_POSTSUBSCRIPT roman_Θ end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_h end_POSTSUPERSCRIPT. As can be seen, the samples from stage-2 are consistent in overall 3D structures but with diverse textures. Thanks to the proposed Point Cloud-structured Latent space, our method supports interactive 3D structure editing. This is achieved by first modifying the stage-1 point cloud 𝐳0,x→𝐳0,x′→subscript𝐳0𝑥superscriptsubscript𝐳0𝑥′{\mathbf{z}}_{0,x}\rightarrow{{\mathbf{z}}_{0,x}^{\prime}}bold_z start_POSTSUBSCRIPT 0 , italic_x end_POSTSUBSCRIPT → bold_z start_POSTSUBSCRIPT 0 , italic_x end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT, and then regenerate the 3D object with the same Gaussian noise.

🔼 This figure demonstrates the benefits of the two-stage cascaded diffusion process and the advantages of latent space editing. Subfigure (a) compares the results of a single-stage diffusion model (generating both geometry and texture at once) with the two-stage approach (geometry first, then texture) from Figure 5. The single-stage method shows inferior texture quality and structural fidelity compared to the cascaded method, highlighting the effectiveness of the two-stage design. Subfigure (b) shows that editing in the point cloud latent space (before decoding to surfel Gaussians) produces cleaner and more realistic results than directly editing the surfel Gaussians themselves. The comparison showcases a reduced chance of introducing artifacts or inconsistencies during the editing process.

read the caption

Figure 7: Qualitative ablation of Cascaded diffusion and latent space editing. We first show the effectiveness of our two-stage cascaded diffusion framework in (a). Compared to Fig. 5, the single-stage 3D diffusion yields worse texture details and 3D structure intactness. In (b), we validate the latent point cloud editing yields less 3D artifacts compared to direct 3D editing on the 3D Gaussians.