Efficient Gaussian Splatting for Monocular Dynamic Scene Rendering via Sparse Time-Variant Attribute Modeling

🔼 This figure illustrates the pipeline of Efficient Dynamic Gaussian Splatting (EDGS). It begins by initializing sparse anchor points from a Structure from Motion (SfM) point cloud using COLMAP. A time-mask Multi-Layer Perceptron (MLP) then classifies each anchor point as either static or dynamic. Static anchors receive k Gaussian offsets, and their time-invariant attributes (opacity, quaternion, scale, color) are determined by small MLPs. Dynamic anchors have their time-variant attributes processed by a deformable attribute MLP, with Gaussian locations at each time step calculated using a Radial Basis Function (RBF) kernel to determine the similarity between each Gaussian and its anchor. The entire process is efficient, using small MLPs for attribute calculation and a single network for deformation. Importantly, anchor positions remain fixed.

read the caption

Figure 2: The pipeline of our EDGS. 1) We first initialize voxelized sparse anchor points from Structure from Motion (SfM) points derived from COLMAP. 2) A time-mask MLP is applied to classify if the anchor belongs to the static area or the deformable area. 3) k𝑘kitalic_k Gaussian offsets are initialized for each anchor 𝒂𝒂\bm{a}bold_italic_a belonging to static area. The time-invariant attributes of each Gaussian, i.e., opacity, quaternion, scale, and color are calculated from its feature by corresponding tiny MLPs. 4) Time-variant attributes for anchors from dynamic areas are decoded by a deformable attribute MLP. RBF kernel function is employed to compute the location of each Gaussian at timestep t𝑡titalic_t by calculating the similarity between each Gaussian and its belonging anchor point. This pipeline is compact and efficient, featuring only a few tiny MLPs for the attributes of the Gaussians and a single network for deformations. Notably, the position of each anchor remains static and is not subject to updates.

🔼 Figure 3 presents a qualitative comparison of novel view synthesis results on the NeRF-DS dataset. The figure shows how different methods, including the authors’ proposed approach and several state-of-the-art (SOTA) techniques, render a scene containing multiple moving objects. The key takeaway is that the authors’ method reconstructs finer details and achieves more structured rendering of moving elements. This is demonstrated by a comparison of how each method renders a cup held in a hand. The improved detail and structure in the authors’ result indicate a more accurate and visually pleasing representation of the dynamic scene.

read the caption

Figure 3: Qualitative comparison on the NeRF-DS dataset (Yan, Li, and Lee 2023). Compared with other SOTA methods, our method reconstructs finer details and produces a structured rendering of the moving objects, e.g., the cup on human’s hand.

🔼 Figure 4 presents a qualitative comparison of dynamic scene reconstruction results on the HyperNeRF dataset. It visually demonstrates the superior performance of the proposed Efficient Dynamic Gaussian Splatting (EDGS) method against other state-of-the-art (SOTA) techniques. EDGS is shown to reconstruct significantly more detailed textures and a more reliable overall scene structure compared to the other methods. This showcases its ability to capture fine details and accurately represent the scene geometry, even in complex dynamic scenarios.

read the caption

Figure 4: Qualitative comparison on the HyperNeRF dataset (Park et al. 2021b). Our EDGS reconstructs detailed texture and reliable structure compared with other SOTA methods.

🔼 Figure 5 compares the rendering results of three different methods: Deformable 3DGS, 4DGS, and the proposed method (Ours). It shows three rows of visualizations for each method, all using a fixed camera perspective across several time steps. The top row displays the rendered frames. The second row presents the difference between each frame and the first frame, highlighting changes over time. The third row illustrates the optical flow, showing movement vectors. The red box emphasizes a region where the static elements appear to exhibit jittering (unwanted movement) in Deformable 3DGS and 4DGS, indicating a deficiency in representing these parts of the scene. The authors’ method avoids this issue, resulting in improved quality for both static and dynamic scene elements.

read the caption

Figure 5: Visuazization of the difference map (diff.) and the optical flow with fixed camera views. We synthesis fixed novel view across time for (Yang et al. 2023; Wu et al. 2023) and ours. The 1s⁢tsuperscript1𝑠𝑡1^{st}1 start_POSTSUPERSCRIPT italic_s italic_t end_POSTSUPERSCRIPT row is the rendered frames at various time steps. The 2n⁢dsuperscript2𝑛𝑑2^{nd}2 start_POSTSUPERSCRIPT italic_n italic_d end_POSTSUPERSCRIPT and 3r⁢dsuperscript3𝑟𝑑3^{rd}3 start_POSTSUPERSCRIPT italic_r italic_d end_POSTSUPERSCRIPT rows are the difference map between tt⁢hsuperscript𝑡𝑡ℎt^{th}italic_t start_POSTSUPERSCRIPT italic_t italic_h end_POSTSUPERSCRIPT frame and the 1s⁢tsuperscript1𝑠𝑡1^{st}1 start_POSTSUPERSCRIPT italic_s italic_t end_POSTSUPERSCRIPT frame and the optical flow, respectively. The response in the highlighted red area indicates that the static area rendered by deformable GS and 4DGS is jittering. Our method achieves better quality for static and dynamic objects.

🔼 Figure 6 visualizes the learned anchor features and the predicted time masks. The visualization employs the UMAP technique to reduce the dimensionality of anchor features for easier interpretation. Two scenes are shown, each with a side and front view. The time mask, generated by the time-mask MLP (Φmask), distinguishes between static (green) and dynamic (red) scene regions by classifying anchor points as static or dynamic based on their time-varying attributes. This highlights the network’s ability to differentiate between these two regions without explicit supervision.

read the caption

Figure 6: Visualization of anchor features and time mask. We visualize the rendered images, anchor features, and time masks for two deformation scenes. Anchor features are visualized using the UMAP function. In the time-mask visualization, the time masks are predicted by time-mask MLP Φm⁢a⁢s⁢ksubscriptΦ𝑚𝑎𝑠𝑘\varPhi_{mask}roman_Φ start_POSTSUBSCRIPT italic_m italic_a italic_s italic_k end_POSTSUBSCRIPT. Green anchors represent static scenes, while red anchors indicate deformation scenes.

🔼 This table presents a quantitative comparison of different novel view synthesis methods on the HyperNeRF’s VRIG dataset. The metrics used for comparison include Peak Signal-to-Noise Ratio (PSNR), Multi-Scale Structural Similarity (MS-SSIM), and Frames Per Second (FPS). The rendering resolution for all methods was 960x540 pixels. The best, second-best, and third-best performing methods for each metric are highlighted with color-coding for easy identification and comparison.

read the caption

Table 2: Quantitative results on HyperNeRF’s (Park et al. 2021b) vrig dataset. The rendering resolution is set to 960×\times×540. We color each cell as best, second best, and third best.

🔼 This table presents an ablation study evaluating the impact of two key components in the proposed Efficient Dynamic Gaussian Splatting (EDGS) method: the anchor-grid initialization strategy and the time-mask Multilayer Perceptron (MLP). The anchor-grid strategy reduces the number of Gaussians needed, which significantly speeds up rendering. The time-mask MLP further enhances speed by identifying and processing only the time-variant (dynamic) anchors, leaving the static anchors unprocessed for better efficiency. The table shows the PSNR (Peak Signal-to-Noise Ratio), a metric for image quality, and the FPS (frames per second), representing the rendering speed, for three scenes under different configurations: using the full model, removing the anchor-grid initialization, and removing the time-mask MLP. The results highlight the contribution of each component to both the speed and quality of the dynamic scene rendering.

read the caption

Table 3: Effects of anchor-voxel strategy and time-mask MLP. The anchor-grid strategy is crucial for rendering speed due to the reduced number of Gaussians. The time-mask MLP filters time-variant anchors, further reducing the number of anchors required for time-variant attribute queries.

🔼 This table compares the performance of four different methods for calculating the changes in Gaussian offset positions over time. The methods are: a rigid transformation approach (assuming all Gaussian offsets move identically to their anchor points); a k-nearest neighbor (KNN) approach; a cosine similarity approach; and a radial basis function (RBF) kernel approach. The table presents the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) for each method across three different scenes from the NeRF-DS dataset, allowing for a comparison of accuracy and quality across varying dynamic scene complexities.

read the caption

Table 4: Gaussian offsets derivation with various strategies. We conduct four different strategies for formulating the variation of the Gaussian offsets across time.