← ArchivePaper2017

Audio-Driven Facial Animation by Joint End-to-End Learning of Pose and Emotion

Tero Karras, Timo Aila, Samuli Laine, Antti Herva, Jaakko Lehtinen

SIGGRAPHIndustrial492 cites9 descendantsFacial Motion Synthesis

End-to-end deep learning system for speech-driven facial animation jointly learning lip sync, head pose, and emotional expression.

Open the paper ↗Trace in the lineage graph

Abstract

We present a machine learning technique for driving 3D facial animation by audio input in real time and with low latency. Our deep neural network learns a mapping from input waveforms to the 3D vertex coordinates of a face model, and simultaneously discovers a compact, latent code that disambiguates the variations in facial expression that cannot be explained by the audio alone. During inference, the latent code can be used as an intuitive control for the emotional state of the face puppet. We train our network with 3--5 minutes of high-quality animation data obtained using traditional, vision-based performance capture methods. Even though our primary goal is to model the speaking style of a single actor, our model yields reasonable results even when driven with audio from other speakers with different gender, accent, or language, as we demonstrate with a user study. The results are applicable to in-game dialogue, low-cost localization, virtual reality avatars, and telepresence.

How to read this

Category: Method: a deep-learning audio-to-face animation system
Contributions: A neural network mapping raw audio waveforms to 3D face-mesh vertex positions in real time at low latency
A jointly learned latent emotion code that disambiguates expression variation not explained by audio and serves as an intuitive control at inference
Demonstrated generalization to other speakers (gender, accent, language) from only 3 to 5 minutes of training animation
Context: Builds on vision-based performance capture used to source training data, in the lineage of Laine et al.'s Production-Level Facial Performance Capture Using Deep CNNs, applying learning to the audio-to-geometry direction.Builds on: Production-Level Facial Performance Capture Using Deep Convolutional Neural Networks
Correctness: Validated for a single actor's speaking style on a small (3 to 5 minute) high-quality dataset with a user study on cross-speaker driving; readers should keep in mind it targets one actor's style and that cross-speaker results are described as reasonable rather than perfect.
Clarity: Accessible; a first pass conveys the joint pose/emotion idea, do a second pass for the network and latent-code formulation.
How to read it: Focus on how the latent emotion code is learned and used as a control; a second pass pays off for the loss design and real-time inference path if you plan to reimplement.

Builds on

Production-Level Facial Performance Capture Using Deep Convolutional Neural Networks 2017

Built upon by

Related work

Keywords

audio-driven facial animation deep neural network convolutional neural network end-to-end learning speech-driven animation latent emotional state performance capture autocorrelation linear predictive coding formant analysis fixed-topology mesh per-vertex deformation pca basis real-time inference visemes coarticulation three-way loss function

This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →