← ArchivePaper2016
Reconstruction of Personalized 3D Face Rigs from Monocular Video
Pablo Garrido, Michael Zollhoefer, Dan Casas, Levi Valgaerts, Kiran Varanasi, Patrick Perez, Christian Theobalt
Builds personalized 3D facial rigs from monocular video, recovering identity-specific blendshapes for downstream animation and editing.
Abstract
We present a novel approach for the automatic creation of a personalized high-quality 3D face rig of an actor from just monocular video data (e.g., vintage movies). Our rig is based on three distinct layers that allow us to model the actor’s facial shape as well as capture his person-specific expression characteristics at high fidelity, ranging from coarse-scale geometry to fine-scale static and transient detail on the scale of folds and wrinkles. At the heart of our approach is a parametric shape prior that encodes the plausible subspace of facial identity and expression variations. Based on this prior, a coarse-scale reconstruction is obtained by means of a novel variational fitting approach. We represent person-specific idiosyncrasies, which cannot be represented in the restricted shape and expression space, by learning a set of medium-scale corrective shapes. Fine-scale skin detail, such as wrinkles, are captured from video via shading-based refinement, and a generative detail formation model is learned. Both the medium- and fine-scale detail layers are coupled with the parametric prior by means of a novel sparse linear regression formulation. Once reconstructed, all layers of the face rig can be conveniently controlled by a low number of blendshape expression parameters, as widely used by animation artists.
How to read this
- Category
- Method: personalized 3D face rig reconstruction from monocular video
- Contributions
- Automatic creation of a high-quality personalized 3D face rig of an actor from only monocular video, including vintage footage
- A three-layer rig: a coarse parametric shape/expression prior fit via a variational approach, learned medium-scale corrective shapes for person-specific idiosyncrasies, and fine-scale wrinkle detail from shading-based refinement with a generative detail model
- A sparse linear regression that couples the medium- and fine-scale detail layers to the parametric prior for downstream animation and editing
- Context
- Builds on parametric face modeling in the lineage of Blanz and Vetter's morphable model, adding corrective and detail layers to produce an editable, identity-specific rig.Builds on: A Morphable Model for the Synthesis of 3D Faces
- Correctness
- Demonstrated on monocular and vintage video; the layered approach explicitly assumes the coarse face fits a restricted shape/expression subspace, with idiosyncrasies and wrinkles recovered as learned corrective and shading-based layers, so fine detail quality depends on the video's lighting and resolution.
- Clarity
- Accessible in structure thanks to the three-layer framing; a first pass conveys the layering, but the variational fit, detail model, and sparse regression reward a second pass.
- How to read it
- First pass for the coarse/medium/fine layered-rig concept and what each layer captures; second pass on the variational fitting and the sparse-regression coupling if you need the reconstruction details.
Builds on
Built upon by
Nothing yet.
Related work
- FaceLab: Scalable Facial Performance Capture for Visual Effects 2020 / DigiPro
- Vdub: Modifying Face Video of Actors for Plausible Visual Alignment to a Dubbed Audio Track 2015 / Eurographics
- It's a UVN Face Rig, Charlie Brown: Facial Techniques for Peanuts 2015 / SIGGRAPH
- Face2Face: Real-Time Face Capture and Reenactment of RGB Videos 2016 / CVPR
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →