← ArchivePaper2013
Realtime Facial Animation with On-the-fly Correctives
Real-time facial animation system that fits corrective blendshapes on-the-fly from depth input to handle fine wrinkle and contact deformation.
Abstract
We introduce a real-time and calibration-free facial performance capture framework based on a sensor with video and depth input. In this framework, we develop an adaptive PCA model using shape correctives that adjust on-the-fly to the actor's expressions through incremental PCA-based learning. Since the fitting of the adaptive model progressively improves during the performance, we do not require an extra capture or training session to build this model. As a result, the system is highly deployable and easy to use: it can faithfully track any individual, starting from just a single face scan of the subject in a neutral pose. Like many real-time methods, we use a linear subspace to cope with incomplete input data and fast motion. To boost the training of our tracking model with reliable samples, we use a well-trained 2D facial feature tracker on the input video and an efficient mesh deformation algorithm to snap the result of the previous step to high frequency details in visible depth map regions. We show that the combination of dense depth maps and texture features around eyes and lips is essential in capturing natural dialogues and nuanced actor-specific emotions. We demonstrate that using an adaptive PCA model not only improves the fitting accuracy for tracking but also increases the expressiveness of the retargeted character.
How to read this
- Category
- Method: real-time facial capture with on-the-fly corrective shapes
- Contributions
- A calibration-free real-time facial performance capture framework using video plus depth input
- An adaptive PCA model whose shape correctives adjust on-the-fly via incremental PCA-based learning, removing the need for a separate training session
- Combining dense depth maps with 2D texture features around eyes and lips, plus mesh deformation, to capture fine details and actor-specific expressions
- Context
- Shares the depth-sensor performance-capture lineage of Weise et al.'s Realtime Performance-Based Facial Animation, extending it with progressively learned corrective blendshapes from a single neutral scan.Builds on: Realtime Performance-Based Facial Animation
- Correctness
- Assumes a single neutral face scan as starting point and that a linear subspace suffices to cope with incomplete/fast input; fine detail depends on visible depth regions and the eye/lip texture features, so accuracy degrades where depth or those cues are weak.
- Clarity
- Readable and well motivated; a first pass conveys the adaptive-corrective idea, a second pass clarifies the incremental PCA and detail-snapping steps.
- How to read it
- Focus on the incremental PCA correctives and how the 2D feature tracker plus depth snapping add detail; a second pass pays off for the fitting pipeline order.
Builds on
Built upon by
Nothing yet.
Related work
- High Fidelity Facial Animation Capture and Retargeting with Contours 2013 / SCA
- Facial Retargeting with Automatic Range of Motion Alignment 2017 / SIGGRAPH
- Avengers: Capturing Thanos's Complex Face 2018 / SIGGRAPH
- Online Modeling for Realtime Facial Animation 2013 / SIGGRAPH
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →