FaceFormer: Speech-Driven 3D Facial Animation with Transformers

How to read this

Category

Method: speech-driven 3D facial animation (Transformer)

Contributions

• A Transformer-based autoregressive model that encodes long-term audio context and predicts a sequence of animated 3D face meshes
• Integrates self-supervised pre-trained speech representations to cope with scarce 3D audio-visual data
• Devises two biased attention mechanisms (biased cross-modal multi-head attention and biased causal self-attention with periodic positional encoding) to align audio-motion and generalize to longer audio

Context

Advances speech-to-3D-face animation beyond short-window phoneme features, building on the speaking-styles capture line of Cudeiro et al.'s VOCA with a Transformer formulation.Builds on: Capture, Learning, and Synthesis of 3D Speaking Styles

Correctness

Assumes long-range audio context and biased attention improve lip accuracy, and leans on pre-trained speech features to offset limited 3D data; claims rest on experiments plus a perceptual user study, so reported superiority is relative to compared baselines and tied to the training corpora used.

Clarity

Accessible if you know Transformers; a first pass conveys the architecture, a second pass for the two biased-attention formulations.

How to read it

First pass for the autoregressive audio-to-mesh framing and why long context matters; second pass on the biased cross-modal and causal attention if you plan to reimplement or adapt the model.