← ArchivePaper2021
Learning an Animatable Detailed 3D Face Model from In-The-Wild Images
DECA reconstructs a detailed animatable 3D face from a single image using a detail-consistency loss to disentangle person-specific wrinkles from expression-dependent deformations.
Abstract
While current monocular 3D face reconstruction methods can recover fine geometric details, they suffer several limitations. Some methods produce faces that cannot be realistically animated because they do not model how wrinkles vary with expression. Other methods are trained on high-quality face scans and do not generalize well to in-the-wild images. We present the first approach that regresses 3D face shape and animatable details that are specific to an individual but change with expression. Our model, DECA (Detailed Expression Capture and Animation), is trained to robustly produce a UV displacement map from a low-dimensional latent representation that consists of person-specific detail parameters and generic expression parameters, while a regressor is trained to predict detail, shape, albedo, expression, pose and illumination parameters from a single image. To enable this, we introduce a novel detail-consistency loss that disentangles person-specific details from expression-dependent wrinkles. This disentanglement allows us to synthesize realistic person-specific wrinkles by controlling expression parameters while keeping person-specific details unchanged. DECA is learned from in-the-wild images with no paired 3D supervision and achieves state-of-the-art shape reconstruction accuracy on two benchmarks.
How to read this
- Category
- Method: learned animatable detailed 3D face model
- Contributions
- DECA regresses 3D face shape plus animatable, person-specific details from a single image, with details that change with expression
- A regressor predicting detail, shape, albedo, expression, pose, and illumination parameters and a UV displacement map from a low-dimensional latent
- A novel detail-consistency loss that disentangles person-specific details from expression-dependent wrinkles, learned from in-the-wild images
- Context
- Builds on the FLAME face model (Li et al., Learning a Model of Facial Shape and Expression from 4D Scans) as its statistical shape basis, extending monocular reconstruction toward expression-driven wrinkle synthesis.Builds on: Learning a Model of Facial Shape and Expression from 4D Scans
- Correctness
- Central assumption is that person-specific detail and expression-dependent wrinkles can be separated via the consistency loss; trained on in-the-wild images for generalization, but readers should remember reconstructed detail is a learned regression, not a measured scan, so fidelity on extreme poses or unusual subjects is not guaranteed.
- Clarity
- Reasonably accessible; a first pass conveys the disentanglement idea, a second pass is needed for the loss formulation and parameter regression.
- How to read it
- Focus on the detail-consistency loss and the shape/detail split in pass one; a second pass on the training setup and loss terms pays off if you intend to reproduce or fine-tune the model.
Built upon by
- EMOCA: Emotion Driven Monocular Face Capture and Animation 2022
- I M Avatar: Implicit Morphable Head Avatars from Videos 2022
- MoRF: Morphable Radiance Fields for Multiview Neural Head Modeling 2022
- Neural Head Avatars from Monocular RGB Videos 2022
- NOFA: NeRF-based One-shot Facial Avatar Reconstruction 2023
- SoftDECA: Computationally Efficient Physics-Based Facial Animations 2023
- SPARK: Self-supervised Personalized Real-time Monocular Face Capture 2024
Related work
- Animatomy: An Animator-Centric, Anatomically Inspired System for 3D Facial Modeling, Animation and Transfer 2022 / SIGGRAPH Asia
- Neural Head Avatars from Monocular RGB Videos 2022 / CVPR
- FLARE: Fast Learning of Animatable and Relightable Mesh Avatars 2023 / SIGGRAPH Asia
- Sketch-Based Controllers for Blendshape Facial Animation 2015 / Eurographics
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →