← ArchivePaper2024
ProbTalk3D: Non-Deterministic Emotion Controllable Speech-Driven 3D Facial Animation Synthesis Using VQ-VAE
Two-stage VQ-VAE learns motion priors; probabilistic codebook sampling produces diverse non-deterministic emotion-controllable facial animations.
Abstract
Audio-driven 3D facial animation synthesis has been an active field of research with attention from both academia and industry. While there are promising results in this area, recent approaches largely focus on lip-sync and identity control, neglecting the role of emotions and emotion control in the generative process. That is mainly due to the lack of emotionally rich facial animation data and algorithms that can synthesize speech animations with emotional expressions at the same time. In addition, majority of the models are deterministic, meaning given the same audio input, they produce the same output motion. We argue that emotions and non-determinism are crucial to generate diverse and emotionally-rich facial animations. In this paper, we propose ProbTalk3D a non-deterministic neural network approach for emotion controllable speech-driven 3D facial animation synthesis using a two-stage VQ-VAE model and an emotionally rich facial animation dataset 3DMEAD. We provide an extensive comparative analysis of our model against the recent 3D facial animation synthesis approaches, by evaluating the results objectively, qualitatively, and with a perceptual user study. We highlight several objective metrics that are more suitable for evaluating stochastic outputs and use both in-the-wild and ground truth data for subjective evaluation.
How to read this
- Category
- Method: speech-driven 3D facial animation synthesis (generative)
- Contributions
- A non-deterministic, emotion-controllable approach to audio-driven 3D facial animation
- A two-stage VQ-VAE that learns motion priors and samples the codebook for diverse outputs
- Comparative analysis (objective, qualitative, and a perceptual user study) against recent methods, using the emotionally rich 3DMEAD dataset
- Context
- Addresses the lip-sync and identity focus of prior transformer-based work such as FaceFormer (Fan 2022) by adding emotion control and non-determinism.Builds on: FaceFormer: Speech-Driven 3D Facial Animation with Transformers
- Correctness
- Validated against recent synthesis methods on the 3DMEAD emotional dataset via objective, qualitative, and perceptual evaluations; results are conditioned on that dataset's emotion coverage, so generalization beyond its expressions and identities should be read with care.
- Clarity
- Reasonably accessible if you know VQ-VAE; a first pass gives the idea, a second pass is needed for the two-stage formulation and sampling scheme.
- How to read it
- First pass for the motivation (emotion plus diversity) and the two-stage architecture; do a second pass on the codebook sampling and the user-study setup if you care about how diversity is measured.
Built upon by
Nothing yet.
Related work
- FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusion 2023 / MIG
- FaceFormer: Speech-Driven 3D Facial Animation with Transformers 2022 / CVPR
- Capture, Learning, and Synthesis of 3D Speaking Styles 2019 / CVPR
- CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior 2023 / CVPR
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →