← ArchivePaper2023

CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior

Jinbo Xing, Menghan Xia, Yuechen Zhang, Xiaodong Cun, Jue Wang, Tien-Tsin Wong

CVPRAcademic269 citesFacial Motion Synthesis

Reformulates speech-driven facial animation as code-query in a learned discrete codebook of realistic facial motion priors, reducing cross-modal mapping uncertainty.

Open the paper ↗Trace in the lineage graph

Abstract

Speech-driven 3D facial animation has been widely studied, yet there is still a gap to achieving realism and vividness due to the highly ill-posed nature and scarcity of audio-visual data. Existing works typically formulate the cross-modal mapping into a regression task, which suffers from the regression-to-mean problem leading to over-smoothed facial motions. In this paper, we propose to cast speech-driven facial animation as a code query task in a finite proxy space of the learned codebook, which effectively promotes the vividness of the generated motions by reducing the cross-modal mapping uncertainty. The codebook is learned by self-reconstruction over real facial motions and thus embedded with realistic facial motion priors. Over the discrete motion space, a temporal autoregressive model is employed to sequentially synthesize facial motions from the input speech signal, which guarantees lip-sync as well as plausible facial expressions. We demonstrate that our approach outperforms current state-of-the-art methods both qualitatively and quantitatively. Also, a user study further justifies our superiority in perceptual quality. Code and video demo are available at https://doubiiu.github.io/projects/codetalker.

How to read this

Category: Method: speech-driven 3D facial animation via a discrete motion prior
Contributions: Recasts speech-driven facial animation as a code-query task over a learned finite codebook instead of a regression task
Learns the codebook by self-reconstruction of real facial motions, embedding realistic facial motion priors
Uses a temporal autoregressive model over the discrete space to synthesize lip-synced, plausible facial motion from speech
Context: Follows transformer-based speech-to-face work such as FaceFormer (Fan et al. 2022), replacing direct regression with a discrete codebook prior.Builds on: FaceFormer: Speech-Driven 3D Facial Animation with Transformers
Correctness: Reported to beat state of the art qualitatively, quantitatively, and in a user study; the approach targets the regression-to-mean over-smoothing problem, but quality is bounded by the codebook expressiveness and the scarce audio-visual data it notes.
Clarity: Readable; the code-query framing is intuitive on a first pass, with a second pass needed for the codebook learning and autoregressive decoding.
How to read it: Focus on why discretization reduces cross-modal uncertainty and how the codebook is learned; a second pass pays off for the autoregressive synthesis and the lip-sync evaluation.

Builds on

FaceFormer: Speech-Driven 3D Facial Animation with Transformers 2022

Built upon by

Nothing yet.

Related work

Keywords

speech-driven 3d facial animation discrete motion prior learned codebook vq-vae vector-quantized autoencoder motion primitives temporal autoregressive model transformer cross-modal mapping lip synchronization wav2vec 2.0 causal self-attention style embedding space flame 3dmm biwi vocaset lip vertex error

This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →