← ArchivePaper2024
3DGS-Avatar: Animatable Avatars via Deformable 3D Gaussian Splatting
Zhiyin Qian, Shaofei Wang, Marko Mihajlovic, Andreas Geiger, Siyu Tang
Learns non-rigid deformation of 3D Gaussians driven by a skeleton with isometric regularization, achieving real-time 50+ FPS rendering and training in 30 minutes from monocular video.
Abstract
We introduce an approach that creates animatable hu-man avatars from monocular videos using 3D Gaussian Splatting (3DGS). Existing methods based on neural radi-ance fields (NeRFs) achieve high-quality novel-viewlnovel-pose image synthesis but often require days of training, and are extremely slow at inference time. Recently, the com-munity has explored fast grid structures for efficient training of clothed avatars. Albeit being extremely fast at training, these methods can barely achieve an interactive ren-de ring frame rate with around 15 FPS. In this paper, we use 3D Gaussian Splatting and learn a non-rigid deformation network to reconstruct animatable clothed human avatars that can be trained within 30 minutes and rendered at real-time frame rates (50+ FPS). Given the explicit nature of our representation, we further introduce as-isometric-as-possible regularizations on both the Gaussian mean vectors and the covariance matrices, enhancing the generalization of our model on highly articulated unseen poses. Experi-mental results show that our method achieves comparable and even better performance compared to state-of-the-art approaches on animatable avatar creation from a monoc-ular input, while being 400x and 250x faster in training and inference, respectively. Please see our project page at https://neuralbodies.github.ioI3DGS-Avatar.
How to read this
- Category
- Method: animatable full-body avatar (deformable 3D Gaussian Splatting)
- Contributions
- Creates animatable clothed human avatars from monocular video using 3D Gaussian Splatting plus a learned non-rigid deformation network driven by a skeleton.
- As-isometric-as-possible regularizations on Gaussian means and covariances to improve generalization to unseen, highly articulated poses.
- Trains within about 30 minutes and renders at real-time rates (50+ FPS), faster than NeRF-based and grid-based predecessors.
- Context
- Positioned against NeRF-based and fast-grid clothed-avatar methods, and builds on forward-skinning ideas for neural shapes (cf. SNARF, Chen et al., 2021).Builds on: SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes
- Correctness
- Reconstructed from monocular video with comparisons reported as comparable or better than state-of-the-art; the isometric regularizers are heuristics to aid pose generalization, and reconstruction quality is bounded by the single-view input and observed pose range.
- Clarity
- Accessible to readers tracking 3DGS avatars; a first pass conveys the deform-network-plus-3DGS design and the speed argument, a second pass for the regularization terms.
- How to read it
- Read the pipeline and the training/inference speed comparison first; do a second pass on the as-isometric-as-possible regularizers if pose generalization is your concern.
Built upon by
Nothing yet.
Related work
- Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies 2021 / CVPR
- PointAvatar: Deformable Point-Based Head Avatars from Videos 2023 / CVPR
- Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans 2021 / CVPR
- SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes 2021 / CVPR
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →