← ArchivePaper2005
Face Transfer with Multilinear Models
Daniel Vlasic, Matthew Brand, Hanspeter Pfister, Jovan Popovic
Multilinear model factoring facial identity and expression for transferring performances between individuals and editing facial animation.
Abstract
Face Transfer maps videorecorded performances of one individual onto facial animations of another by extracting visemes, expressions, and 3D pose from monocular video. It builds on a multilinear model of 3D face meshes that separably parameterizes geometric variation due to identity, expression, and viseme, estimated from a Cartesian product of 3D face scans using N-mode SVD. The paper introduces methods to put unstructured scans into correspondence via template fitting and to impute missing examples in the data tensor through matrix factorization. By linking the multilinear model to optical-flow-based tracking with a weak-perspective camera model, the system recovers pose and attribute parameters from video and can mix attributes across multiple videos to rewrite footage or retarget performances to new identities.
How to read this
- Category
- Method: a multilinear face model for performance transfer and editing
- Contributions
- A multilinear 3D face model that separably parameterizes identity, expression, and viseme, estimated via N-mode SVD over a Cartesian product of face scans
- Techniques to put unstructured scans into correspondence (template fitting) and impute missing tensor examples via matrix factorization
- A system linking the model to optical-flow tracking with a weak-perspective camera to recover pose and attributes from monocular video, then retarget or mix performances across identities
- Context
- Extends the linear morphable-model line (Blanz and Vetter, A Morphable Model for the Synthesis of 3D Faces) to a multilinear tensor factorization that disentangles multiple modes of facial variation rather than a single PCA space.Builds on: A Morphable Model for the Synthesis of 3D Faces
- Correctness
- Demonstrated on monocular video tracking and cross-subject retargeting, but it assumes scans can be brought into reliable correspondence and a weak-perspective camera; recovery quality depends on optical flow and on how well the scan corpus spans the target identities and expressions.
- Clarity
- Reasonably accessible at a high level; a first pass conveys the factor-then-track idea, a second pass is needed for the N-mode SVD formulation and the tracking objective.
- How to read it
- First pass to grasp the identity/expression/viseme factorization and the video-to-attributes pipeline; do a second pass on the N-mode SVD and correspondence/imputation steps if you intend to reimplement or adapt the model.
Builds on
Related work
- Performance-Driven Facial Animation 1990 / SIGGRAPH
- Vision-based control of 3D facial animation 2003 / SCA
- Transferring the Rig and Animations from a Character to Different Face Models 2008 / CGF
- Realtime Performance-Based Facial Animation 2011 / SIGGRAPH
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →