← ArchivePaper2013

3D Shape Regression for Real-Time Facial Animation

Chen Cao, Yanlin Weng, Stephen Lin, Kun Zhou

SIGGRAPHAcademic381 citesFacial

Real-time 3D face shape regression from RGB video, tracking both rigid head pose and non-rigid expression deformation simultaneously.

Open the paper ↗Trace in the lineage graph

Abstract

We present a real-time performance-driven facial animation system based on 3D shape regression. In this system, the 3D positions of facial landmark points are inferred by a regressor from 2D video frames of an ordinary web camera. From these 3D points, the pose and expressions of the face are recovered by fitting a user-specific blendshape model to them. The main technical contribution of this work is the 3D regression algorithm that learns an accurate, user-specific face alignment model from an easily acquired set of training data, generated from images of the user performing a sequence of predefined facial poses and expressions. Experiments show that our system can accurately recover 3D face shapes even for fast motions, non-frontal faces, and exaggerated expressions. In addition, some capacity to handle partial occlusions and changing lighting conditions is demonstrated.

How to read this

Category: Method / system: real-time performance-driven facial animation
Contributions: Real-time facial animation from an ordinary webcam via 3D shape regression that infers 3D landmark positions from 2D frames
Recovers head pose and expression by fitting a user-specific blendshape model to the regressed 3D points
Learns the user-specific alignment model from an easily acquired set of predefined poses and expressions
Context: Rests on the morphable / blendshape face-model tradition (Blanz and Vetter A Morphable Model), pairing it with a learned 3D regressor for monocular tracking.Builds on: A Morphable Model for the Synthesis of 3D Faces
Correctness: Shown to recover shapes for fast motion, non-frontal faces, and exaggerated expressions with some tolerance to partial occlusion and lighting change; it requires a per-user training/calibration session and accuracy is tied to that user-specific model.
Clarity: Accessible system description; a first pass conveys the regress-then-fit pipeline, a second pass details the regressor training.
How to read it: First read the pipeline (2D to 3D regression, then blendshape fit) and the calibration step; a second pass on the regression training pays off if you care about tracking robustness or want to reproduce it.

Builds on

A Morphable Model for the Synthesis of 3D Faces 1999

Built upon by

Nothing yet.

Related work

FaceWarehouse: A 3D Facial Expression Database for Visual Computing 2014 / TVCG
Production-Level Facial Performance Capture Using Deep Convolutional Neural Networks 2017 / SCA
Face Transfer with Multilinear Models 2005 / SIGGRAPH
Vision-based control of 3D facial animation 2003 / SCA

Keywords

3d shape regression real-time facial animation performance-driven animation user-specific blendshapes facial landmarks face tracking monocular video tracking boosted regression fern facewarehouse bilinear face model facial action coding system pose space deformation 3d avatars expression coefficients camera calibration data augmentation

This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →