← ArchivePaper2014
Displaced Dynamic Expression Regression for Real-Time Facial Tracking and Animation
Regresses displaced expression shapes from RGB video enabling real-time facial tracking and animation with detailed wrinkle representation.
Abstract
We present a fully automatic approach to real-time facial tracking and animation with a single video camera. Our approach does not need any calibration for each individual user. It learns a generic regressor from public image datasets, which can be applied to any user and arbitrary video cameras to infer accurate 2D facial landmarks as well as the 3D facial shape from 2D video frames. The inferred 2D landmarks are then used to adapt the camera matrix and the user identity to better match the facial expressions of the current user. The regression and adaptation are performed in an alternating manner. With more and more facial expressions observed in the video, the whole process converges quickly with accurate facial tracking and animation. In experiments, our approach demonstrates a level of robustness and accuracy on par with state-of-the-art techniques that require a time-consuming calibration step for each individual user, while running at 28 fps on average. We consider our approach to be an attractive solution for wide deployment in consumer-level applications.
How to read this
- Category
- Method: regression-based real-time facial tracking from RGB video
- Contributions
- A fully automatic, calibration-free real-time facial tracking and animation system using a single video camera
- A generic regressor learned from public image datasets that infers 2D landmarks and 3D facial shape, applicable to any user and camera
- Alternating regression and adaptation of camera matrix and user identity, running at around 28 fps
- Context
- Uses the FaceWarehouse 3D expression database (Cao et al.) as the basis for learning the generic regressor and shape priors.Builds on: FaceWarehouse: A 3D Facial Expression Database for Visual Computing
- Correctness
- Assumes a generic regressor trained on public datasets transfers to arbitrary users/cameras and that alternating adaptation converges; reported to match calibration-based methods in robustness/accuracy while running real-time, so reader should note accuracy depends on RGB-only cues without depth.
- Clarity
- Accessible; a first pass conveys the regress-then-adapt loop, a second pass is needed for the displaced-expression regression formulation.
- How to read it
- Focus on what the regressor predicts and how the alternating identity/camera adaptation refines it; a second pass pays off for the displaced dynamic expression regression details.
Built upon by
Related work
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →