← ArchivePaper2011
High-Quality Passive Facial Performance Capture Using Anchor Frames
Thabo Beeler, Fabian Hahn, Derek Bradley, Bernd Bickel, Paul Beardsley, Craig Gotsman, Robert Sumner, Markus Gross
Passive multi-view facial performance capture using anchor frames for temporal coherence, achieving high-quality reconstruction without active illumination.
Abstract
We present a new technique for passive and markerless facial performance capture based on anchor frames. Our method starts with high resolution per-frame geometry acquisition using state-of-the-art stereo reconstruction, and proceeds to establish a single triangle mesh that is propagated through the entire performance. Leveraging the fact that facial performances often contain repetitive subsequences, we identify anchor frames as those which contain similar facial expressions to a manually chosen reference expression. Anchor frames are automatically computed over one or even multiple performances. We introduce a robust image-space tracking method that computes pixel matches directly from the reference frame to all anchor frames, and thereby to the remaining frames in the sequence via sequential matching. This allows us to propagate one reconstructed frame to an entire sequence in parallel, in contrast to previous sequential methods. Our anchored reconstruction approach also limits tracker drift and robustly handles occlusions and motion blur. The parallel tracking and mesh propagation offer low computation times. Our technique will even automatically match anchor frames across different sequences captured on different occasions, propagating a single mesh to all performances.
How to read this
- Category
- Capture system / method: passive markerless facial performance capture
- Contributions
- Passive, markerless facial performance capture based on anchor frames, propagating a single triangle mesh through an entire performance
- A robust image-space tracking method computing pixel matches directly from a reference frame to anchor frames, then to remaining frames via sequential matching, enabling parallel propagation
- Anchored reconstruction that limits tracker drift, handles occlusions and motion blur, and can match anchor frames across multiple performances at low computation time
- Context
- Builds on the authors' single-shot facial geometry capture (Beeler et al. 2010) and state-of-the-art stereo reconstruction, extending per-frame geometry into temporally coherent performance capture.Builds on: High-Quality Single-Shot Capture of Facial Geometry
- Correctness
- Demonstrated on passive multi-view facial performances; the method assumes performances contain repetitive subsequences (similar expressions) so suitable anchor frames exist, and relies on a manually chosen reference expression, which a reader should keep in mind for sparse or highly varied performances.
- Clarity
- Clearly motivated and well-structured; a first pass conveys the anchor-frame idea and pipeline, with a second pass for the image-space tracking and matching details.
- How to read it
- First pass to grasp anchor frames and why parallel propagation beats sequential tracking; second pass on the tracking/matching formulation if you care about drift handling or reimplementation.
Builds on
Built upon by
- Facial Performance Enhancement Using Dynamic Shape Space Analysis 2014
- An Anatomically Constrained Local Deformation Model for Monocular Face Capture 2016
- Facial Performance Capture for Digital Doubles: Quantum Break and Agent Six 2016
- Masquerade: Fine-Scale Details for Head-Mounted Camera Motion Capture Data 2017
- An Empirical Rig for Jaw Animation 2018
- Practical Person-Specific Eye Rigging 2019
- FaceLab: Scalable Facial Performance Capture for Visual Effects 2020
Related work
- High Resolution Passive Facial Performance Capture 2010 / SIGGRAPH
- Vision-based control of 3D facial animation 2003 / SCA
- Driving High-Resolution Facial Scans with Video Performance Capture 2015 / SIGGRAPH
- FaceLab: Scalable Facial Performance Capture for Visual Effects 2020 / DigiPro
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →