← ArchivePaper2015
Real-Time High-Fidelity Facial Performance Capture
Real-time system for capturing high-fidelity facial performance from RGB cameras using a personalized blendshape model.
Abstract
We present the first real-time high-fidelity facial capture method. The core idea is to enhance a global real-time face tracker, which provides a low-resolution face mesh, with local regressors that add in medium-scale details, such as expression wrinkles. Our main observation is that although wrinkles appear in different scales and at different locations on the face, they are locally very self-similar and their visual appearance is a direct consequence of their local shape. We therefore train local regressors from high-resolution capture data in order to predict the local geometry from local appearance at runtime. We propose an automatic way to detect and align the local patches required to train the regressors and run them efficiently in real-time. Our formulation is particularly designed to enhance the low-resolution global tracker with exactly the missing expression frequencies, avoiding superimposing spatial frequencies in the result. Our system is generic and can be applied to any real-time tracker that uses a global prior, e.g. blend-shapes. Once trained, our online capture approach can be applied to any new user without additional training, resulting in high-fidelity facial performance reconstruction with person-specific wrinkle details from a monocular video camera in real-time.
How to read this
- Category
- Capture system: real-time high-fidelity facial capture
- Contributions
- Presents a real-time high-fidelity facial capture method that augments a low-resolution global face tracker with medium-scale detail such as expression wrinkles.
- Trains local regressors on high-resolution capture data to predict local geometry from local appearance at runtime, exploiting the self-similarity of wrinkles.
- Formulated to add exactly the frequencies missing from the global tracker (avoiding superimposed frequencies), is tracker-agnostic, and generalizes to new users without extra training.
- Context
- Extends real-time blendshape-based facial tracking, building directly on Cao et al.'s Displaced Dynamic Expression Regression by adding a local-detail regression layer.Builds on: Displaced Dynamic Expression Regression for Real-Time Facial Tracking and Animation
- Correctness
- Relies on the assumption that wrinkle appearance is locally self-similar and a direct consequence of local shape; demonstrated on RGB camera input with a personalized blendshape model, so detail fidelity depends on the training data and the global tracker's quality.
- Clarity
- Accessible in concept (global tracker plus local detail); a first pass conveys the idea, a second pass for the patch detection/alignment and regressor training.
- How to read it
- First pass for the global-plus-local-detail decomposition and the missing-frequency argument; second pass on the local regressor training if reproducing the detail layer.
Related work
- FaceScape: A Large-Scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction 2020 / CVPR
- Driving High-Resolution Facial Scans with Video Performance Capture 2015 / SIGGRAPH
- FaceWarehouse: A 3D Facial Expression Database for Visual Computing 2014 / TVCG
- Dynamic 3D Avatar Creation from Hand-Held Video Input 2015 / SIGGRAPH
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →