Production-Level Facial Performance Capture Using Deep Convolutional Neural Networks

How to read this

Category

Method / system: deep CNN for video-based facial performance capture

Contributions

• A real-time framework that densely tracks an actor's face from monocular video, predicting per-frame positions of roughly 5000 mesh vertices from a single grayscale image
• Use of a high-end multi-view stereo and artist-cleanup pipeline on a few minutes of footage to bootstrap training data, with a PCA-initialized output layer and data augmentation
• Automatic processing of the remaining footage at high frame rates, with plausible inference in self-occluded regions such as eyes and lips

Context

Follows the real-time high-fidelity facial capture line (Cao et al., Real-Time High-Fidelity Facial Performance Capture), replacing hand-crafted tracking with a learned CNN regressor.Builds on: Real-Time High-Fidelity Facial Performance Capture

Correctness

The network is trained per actor from that actor's own clean-up-quality footage, so accuracy is tied to the bootstrap pipeline and the trained subject; readers should treat the throughput and quality as demonstrated within that production setup rather than as a general cross-actor model.

Clarity

Clear and practical; a first pass conveys the bootstrap-then-infer idea, a second pass covers the network and PCA-output details.

How to read it

Focus on the data-bootstrapping strategy and PCA-initialized output that make the CNN production-viable; a second pass on architecture and augmentation pays off if you build a similar capture system.