Realtime Performance-Based Facial Animation

How to read this

Category

System / method: real-time performance-based facial animation from a depth sensor

Contributions

• A system letting any user drive a digital avatar's facial expressions in real time, recorded with a non-intrusive commercial 3D sensor in a natural environment
• A face tracking algorithm combining geometry and texture registration with pre-recorded animation priors in a single optimization, formulated as MAP estimation in a reduced parameter space
• Markerless, lighting-free, hardware-light real-time reconstruction of compelling 3D facial dynamics, enabling gameplay and social-interaction applications

Context

Builds on example-based facial rigging (Li et al. 2010) and blendshape/animation-prior models, adapting them to noisy consumer depth-camera input for live retargeting.Builds on: Example-Based Facial Rigging

Correctness

Demonstrated with a commodity 3D sensor; the method assumes pre-recorded animation priors and a reduced expression space to compensate for high sensor noise, so output quality is bounded by the prior and the per-user model rather than capturing arbitrary unseen expressions.

Clarity

Accessible and application-driven; a first pass conveys the live-avatar idea and pipeline, with a second pass for the MAP optimization and the registration/prior terms.

How to read it

First pass for the real-time markerless concept and where the animation priors fit; second pass on the single-optimization formulation if you want the tracking math or to reproduce the stability behavior.