TaoAvatar: Real-Time Lifelike Full-Body Talking Avatars for Augmented Reality via 3D Gaussian Splatting

🔼 This figure illustrates the three main stages of the TaoAvatar method. (a) shows the creation of a personalized clothed human model (SMPLX++) with Gaussian textures bound to its surface. (b) details the training of a StyleUnet (teacher network) to learn complex, pose-dependent non-rigid deformations. These deformations are then distilled into a simpler MLP (student network) for efficient real-time processing. (c) shows how learnable Gaussian blend shapes are added to further refine the avatar’s appearance, resulting in high-fidelity rendering.

read the caption

Figure 2: Illustration of our Method. Our pipeline begins by reconstructing (a) a cloth-extended SMPLX mesh with aligned Gaussian textures. To address complex dynamic non-rigid deformations, (b) we employ a teacher StyleUnet to learn pose-dependent non-rigid maps, which are then baked into a lightweight student MLP to infer non-rigid deformations of our template mesh. For high-fidelity rendering, (c) we introduce learnable Gaussian blend shapes to enhance appearance details.

🔼 This figure presents a qualitative comparison of full-body talking avatar generation results across different methods, including the authors’ proposed method (Ours) and several state-of-the-art baselines. Each row represents a different pose and expression from a single video sequence, showcasing the performance of each approach. The comparison highlights the superiority of the authors’ method in capturing the fine details of the clothing, including natural folds and wrinkles, and in rendering more realistic and expressive facial features. This visual comparison underscores the effectiveness of the proposed approach in achieving high-fidelity, real-time avatar generation.

read the caption

Figure 3: Qualitative comparisons on full-body talking tasks. Our method outperforms state-of-the-art methods by producing clearer clothing dynamics and enhanced facial details.

🔼 Figure 4 showcases the robustness of the proposed method in handling challenging scenarios. It presents several examples of avatar reconstructions under extreme poses and exaggerated facial expressions, highlighting the method’s ability to maintain high-quality rendering even in these difficult conditions. The images visually demonstrate the superior performance of the TaoAvatar model compared to existing techniques in accurately representing detailed facial features and clothing dynamics, irrespective of complex body movements.

read the caption

Figure 4: Results in challenging scenarios. Our method can obtain high-quality reconstruction even for challenging pose and expressions.

🔼 This figure showcases TaoAvatar’s capability to generate realistic animations using only skeleton and expression parameters. Different individuals are shown performing the same actions, highlighting the model’s ability to capture and reproduce nuanced movements and expressions consistently across various subjects.

read the caption

Figure 5: Novel pose and expression animation. TaoAvatar can be driven by the same skeleton and expression parameters vividly.

🔼 This ablation study visualizes the impact of different components of the proposed method on the final avatar quality. By comparing the results with and without certain modules (like mesh non-rigid deformation or Gaussian non-rigid deformation), the figure highlights their individual contributions to generating high-fidelity avatars. Red and green boxes specifically point out artifacts present in the results and how these artifacts are addressed in the improved versions, providing a clear visual demonstration of the effectiveness of each component.

read the caption

Figure 6: Ablation Study. Red and Green boxes show artifacts and their improved counterparts.

🔼 This figure illustrates the process of reconstructing a personalized clothed human model, called SMPLX++, which serves as the foundation for creating the 3D avatar. The pipeline begins with multi-view images of a person. These images are used to reconstruct a 3D mesh using NeuS2, a neural radiance field method. Next, the non-body components (like clothing and hair) are segmented and separated from the body mesh. The SMPLX model (a parametric human body model) parameters are estimated for a reference frame (usually a T-pose) and the skinning weights are propagated from the body mesh to the non-body components. Inverse skinning is then applied to transform the non-body components back to the reference pose. Finally, the body mesh and non-body components are combined to create the complete clothed parametric model SMPLX++.

read the caption

Figure 7: The Pipeline of the Template SMPLX++ Reconstruction.

🔼 Figure 8 displays a comparison of different template meshes used to represent human avatars. It showcases the original SMPL-X model, the MeshAvatar template, the AnimatableGS template, and the novel SMPLX++ template developed by the authors. The visual differences highlight the improvements in detail and clothing representation achieved by the SMPLX++ template, which forms the foundation of the authors’ proposed method for creating high-fidelity, real-time avatars.

read the caption

Figure 8: Template Comparison.

🔼 This figure shows the relighting visualization process. The input is a raw rendered image, and the method uses the rendered normal map and an environment map to perform image-based relighting. The resulting image is more realistic and integrates better with the surrounding environment.

read the caption

Figure 9: Relighting Visualisation.

🔼 This figure details the architecture of the mesh non-rigid deformation field, a key component in TaoAvatar’s system for creating lifelike full-body avatars. It showcases the network’s input (pose and vertex coordinates), the processing steps involving two separate MLPs (Multilayer Perceptrons) to handle body and clothing deformations separately and the final output (the non-rigid deformation). The inclusion of a ‘cloth mask’ highlights the method’s ability to specifically apply clothing deformation where needed.

read the caption

Figure 10: Network Architecture of Mesh Nonrigid Deformation Field.

🔼 This figure shows an ablation study on the effect of adding a normal loss during the training process. The normal loss helps to improve the accuracy of the rendered normal maps by promoting smoother and more realistic-looking normal vectors. The comparison shows a significant improvement in the quality of the normals when the normal loss is included, resulting in better rendering quality and realism.

read the caption

Figure 11: The Impact of Normal Loss.

🔼 This figure shows a qualitative comparison of the results obtained from the teacher and student networks during the ‘baking’ process. The images compare the ground truth, the teacher network output, and the student network output. It visually demonstrates how the student network, a lightweight model, effectively learns and reproduces the complex non-rigid deformations initially learned by the more computationally expensive teacher network. The comparison shows the accuracy of the student network in approximating the deformations from the teacher network across different poses and includes a detailed look at the generated semantic maps (Sem. Et, Sem. Es) and Gaussian non-rigid deformation maps (Gau. Non. ∆Uf, Mesh Non. ∆V).

read the caption

Figure 12: Qualitative Visualization of Baking.

🔼 This figure illustrates the complete pipeline for creating a 3D digital human agent that can interact in real time on Apple Vision Pro. It shows how user audio input is processed through Automatic Speech Recognition (ASR), converted into text, and then interpreted by a large language model (LLM). The LLM’s response generates text, which is then converted to speech via Text-to-Speech (TTS) and finally used to drive the facial expressions and body movements of the 3D avatar using a Body Motion Library and Audio2BS.

read the caption

Figure 13: 3D Digital Human Agent Pipeline.

🔼 This ablation study visualizes the effect of including the semantic loss (Lsem) during the baking process of the neural network. It shows a comparison of the rendered images with and without the semantic loss. The images demonstrate the improvements in detail and accuracy achieved by incorporating Lsem, particularly in areas where clothing and body intersect. The results highlight the importance of Lsem for resolving inconsistencies in the Gaussian splatting representation.

read the caption

Figure 14: Ablation Study on Semantic Loss during Baking.

🔼 This figure provides a detailed qualitative comparison of the results produced by different methods for generating talking avatars. It showcases the level of detail achieved in face regions by comparing the results of the teacher model, the student model, and other state-of-the-art methods such as GaussianAvatar, MeshAvatar, 3DGS-Avatar, and AnimatableGS against the ground truth. This visual comparison highlights the superior detail achieved by the proposed model, especially in terms of facial expressions and textural details.

read the caption

Figure 15: Qualitative Comparison of Details.

🔼 This table presents a quantitative comparison of different methods for reconstructing human avatars performing complex motions and exaggerated expressions. The metrics used are PSNR, SSIM, and LPIPS, evaluating the quality of the rendered images. Results are shown for the full body and, separately, for the face (particularly focusing on exaggerated expressions). Higher PSNR and SSIM values and lower LPIPS values indicate better reconstruction quality.

read the caption

Table 2: Quantitative comparisons about complex motions and expressions reconstruction. The results inside the parentheses are evaluated for face regions with exaggerated expressions.

🔼 This table presents the results of an ablation study conducted to analyze the impact of different components on the overall performance of the proposed method. The study focuses on evaluating the quality of the reconstructed mesh geometry by comparing it to a pseudo-ground truth mesh generated using NeuS2 [54]. Different configurations are tested, systematically removing or modifying certain elements to assess their contribution to the final output. The results highlight the importance of each component, helping determine the effectiveness and robustness of the approach.

read the caption

Table 3: Ablation Study. We use the mesh reconstructed by NeuS2 [54] as a pseudo-truth for geometry evaluation.

🔼 Table 4 provides a comparison of several state-of-the-art methods for generating full-body avatars. It contrasts different aspects of each method, including the type of template used (SMPLX or a customized version), the method used for non-rigid deformation modeling (MLP, StyleUnet), the number of Gaussians or parameters used, the overall quality of the resulting avatar, the level of controllability afforded to the user, the complexity of the body representation, and the speed of inference. This allows for a quantitative assessment of the trade-offs between accuracy, controllability, realism, and computational efficiency in different approaches.

read the caption

Table 4: Summary about these State-of-the-art Methods of Full-body Avatars.