Face2Face: Real-Time Face Capture and Reenactment of RGB Videos

How to read this

Category

Method: real-time monocular face reenactment system

Contributions

• Real-time expression transfer from a webcam source actor onto a monocular target video, re-rendered photo-realistically
• Dense analysis-by-synthesis tracking over a multilinear PCA face prior, with non-rigid model-based bundling to recover identity from monocular video
• Subspace deformation transfer in blendshape space plus a data-driven mouth-interior synthesis that retrieves and warps the best matching target mouth frame

Context

Builds on parametric face modeling in the lineage of Blanz and Vetter's morphable model, extending it to live monocular tracking and reenactment.Builds on: A Morphable Model for the Synthesis of 3D Faces

Correctness

Demonstrated on commodity webcam input and existing monocular videos; the realism leans on the multilinear prior and the mouth-retrieval step, so readers should keep in mind it assumes a recoverable target mouth appearance and a face that fits the learned subspace.

Clarity

Accessible at a high level; a first pass conveys the pipeline and the reenactment idea, but a second pass is needed for the analysis-by-synthesis optimization and the IRLS solver.

How to read it

First pass for the four-stage pipeline (tracking, bundling, transfer, mouth synthesis); do a second pass on the photometric energy formulation and GPU IRLS solver if you care about the real-time tracking math.