← ArchivePaper2022
Neural Head Avatars from Monocular RGB Videos
Philip-William Grassal, Malte Prinzler, Titus Leistner, Carsten Rother, Matthias Niessner, Justus Thies
Hybrid avatar representation combining a coarse morphable face model with two networks predicting mesh vertex offsets and view- and expression-dependent textures.
Abstract
We present Neural Head Avatars, a novel neural representation that explicitly models the surface geometry and appearance of an animatable human avatar that can be used for teleconferencing in AR/VR or other applications in the movie or games industry that rely on a digital human.11philgras.github.io/neural_head_avatars/neural_head_avatars.html Our representation can be learned from a monocular RGB portrait video that features a range of different expressions and views. Specifically, we propose a hybrid representation consisting of a morphable model for the coarse shape and expressions of the face, and two feed-forward networks, predicting vertex offsets of the underlying mesh as well as a view- and expression-dependent texture. We demonstrate that this representation is able to accurately extrapolate to unseen poses and view points, and generates natural expressions while providing sharp texture details. Compared to previous works on head avatars, our method provides a disentangled shape and appearance model of the complete human head (including hair) that is compatible with the standard graphics pipeline. Moreover, it quantitatively and qualitatively outperforms current state of the art in terms of reconstruction quality and novel-view synthesis.
How to read this
- Category
- Method: animatable neural head avatar from monocular video
- Contributions
- A hybrid head representation combining a coarse morphable face model with two feed-forward networks for mesh vertex offsets and a view- and expression-dependent texture
- Learns a complete head avatar (including hair) from a single monocular RGB portrait video
- Produces a disentangled shape/appearance model compatible with the standard graphics pipeline that extrapolates to unseen poses and viewpoints
- Context
- Builds on monocular morphable-model face reconstruction such as Feng et al. DECA (2021), adding learned geometry offsets and neural texture for a full animatable head.Builds on: Learning an Animatable Detailed 3D Face Model from In-The-Wild Images
- Correctness
- Reported to quantitatively and qualitatively outperform prior head-avatar work and extrapolate to unseen poses/views; it relies on a morphable-model prior and a single subject's monocular video, so coverage of expressions and views in that video bounds quality.
- Clarity
- Accessible; a first pass conveys the hybrid (model + two networks) design, a second pass covers the offset and texture network formulations.
- How to read it
- First pass for the morphable-model-plus-networks hybrid and why it stays graphics-pipeline compatible; second pass on the vertex-offset and view/expression texture networks if reconstructing avatars.
Related work
- PointAvatar: Deformable Point-Based Head Avatars from Videos 2023 / CVPR
- Learning an Animatable Detailed 3D Face Model from In-The-Wild Images 2021 / SIGGRAPH
- Reconstruction of Personalized 3D Face Rigs from Monocular Video 2016 / SIGGRAPH
- FLARE: Fast Learning of Animatable and Relightable Mesh Avatars 2023 / SIGGRAPH Asia
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →