← ArchivePaper2021
Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans
Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, Xiaowei Zhou
Anchors per-frame neural radiance latent codes to a deformable SMPL mesh so that sparse-view observations are integrated across time for dynamic human reconstruction.
Abstract
This paper addresses the challenge of novel view synthesis for a human performer from a very sparse set of camera views. Some recent works have shown that learning implicit neural representations of 3D scenes achieves remarkable view synthesis quality given dense input views. However, the representation learning will be ill-posed if the views are highly sparse. To solve this ill-posed problem, our key idea is to integrate observations over video frames. To this end, we propose Neural Body, a new human body representation which assumes that the learned neural representations at different frames share the same set of latent codes anchored to a deformable mesh, so that the observations across frames can be naturally integrated. The deformable mesh also provides geometric guidance for the network to learn 3D representations more efficiently. To evaluate our approach, we create a multi-view dataset named ZJU-MoCap that captures performers with complex motions. Experiments on ZJU-MoCap show that our approach outperforms prior works by a large margin in terms of novel view synthesis quality. We also demonstrate the capability of our approach to reconstruct a moving person from a monocular video on the People-Snapshot dataset.
How to read this
- Category
- Method: implicit neural representation for dynamic human view synthesis
- Contributions
- Neural Body, a human representation where per-frame neural fields share one set of latent codes anchored to a deformable mesh, integrating sparse-view observations across video frames
- Use of the deformable mesh as geometric guidance to learn 3D representations more efficiently from highly sparse views
- The ZJU-MoCap multi-view dataset of performers with complex motions, used to show large improvement in novel-view-synthesis quality over prior work
- Context
- Combines neural-radiance-field-style implicit scene representations with parametric body modelling via SMPL (Loper et al.), anchoring structured latent codes to the SMPL mesh to make sparse-view dynamic reconstruction well posed.Builds on: SMPL: A Skinned Multi-Person Linear Model
- Correctness
- Key assumption is that frames share latent codes anchored to an accurate deformable body mesh, so quality depends on body-fit accuracy and on motion being captured across frames; evaluated on the authors' ZJU-MoCap data, meaning readers should weigh results against that capture setting and the monocular case shown as a demonstration.
- Clarity
- Accessible idea (codes anchored to a mesh); a first pass conveys the integrate-over-frames insight, a second pass clarifies the latent-code structure and rendering.
- How to read it
- Focus pass one on the structured-latent-code-on-mesh idea and why it resolves the sparse-view ill-posedness; a second pass on the network and dataset pays off if you plan to reproduce or benchmark on ZJU-MoCap.
Builds on
Built upon by
Related work
- Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies 2021 / CVPR
- 3DGS-Avatar: Animatable Avatars via Deformable 3D Gaussian Splatting 2024 / CVPR
- S3: Neural Shape, Skeleton, and Skinning Fields for 3D Human Modeling 2021 / CVPR
- SMPLicit: Topology-aware Generative Model for Clothed People 2021 / CVPR
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →