← ArchivePaper2023

FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusion

Stefan Stan, Kazi Injamamul Haque, Zerrin Yumak

MIGAcademic120 citesFacial Motion Synthesis

Diffusion model for generating diverse and expressive 3D facial animations driven by speech audio input.

Open the paper ↗Trace in the lineage graph

Abstract

Speech-driven 3D facial animation synthesis has been a challenging task both in industry and research. Recent methods mostly focus on deterministic deep learning methods meaning that given a speech input, the output is always the same. However, in reality, the non-verbal facial cues that reside throughout the face are non-deterministic in nature. In addition, majority of the approaches focus on 3D vertex based datasets and methods that are compatible with existing facial animation pipelines with rigged characters is scarce. To eliminate these issues, we present FaceDiffuser, a non-deterministic deep learning model to generate speech-driven facial animations that is trained with both 3D vertex and blendshape based datasets. Our method is based on the diffusion technique and uses the pre-trained large speech representation model HuBERT to encode the audio input. To the best of our knowledge, we are the first to employ the diffusion method for the task of speech-driven 3D facial animation synthesis. We have run extensive objective and subjective analyses and show that our approach achieves better or comparable results in comparison to the state-of-the-art methods. We also introduce a new in-house dataset that is based on a blendshape based rigged character. The code and the dataset will be publicly available on the project page1.

How to read this

Category: Method: speech-driven 3D facial animation (diffusion)
Contributions: FaceDiffuser, a non-deterministic diffusion model for speech-driven 3D facial animation
Trained on both 3D vertex and blendshape datasets, targeting compatibility with rigged-character pipelines
Uses a pretrained HuBERT speech encoder; reports objective and subjective evaluation versus state of the art
Context: Builds on speech-driven facial animation work such as Karras et al. (karras-audio-driven-2017) and FaceFormer (faceformer-fan-2022), reframing the task with diffusion to capture non-deterministic facial cues.Builds on: Audio-Driven Facial Animation by Joint End-to-End Learning of Pose and Emotion · FaceFormer: Speech-Driven 3D Facial Animation with Transformers
Correctness: The key premise is that facial motion is non-deterministic, so diffusion is appropriate; results are reported as better or comparable to prior work, meaning gains are partly about diversity rather than uniformly higher accuracy, and depend on the training datasets used.
Clarity: Accessible; a first pass conveys the deterministic-versus-stochastic motivation and the HuBERT-plus-diffusion setup, a second pass for the training and evaluation details.
How to read it: Focus on why non-determinism matters here and on the blendshape compatibility angle; a second pass is worth it to see how the objective and subjective results are balanced.

Builds on

Built upon by

Nothing yet.

Related work

Keywords

speech-driven facial animation diffusion model denoising hubert blendshapes vertex displacement facial controls gru decoder non-deterministic synthesis lip vertex error facial dynamics deviation wav2vec 2.0 rigged characters biwi dataset vocaset mesh animation diversity metric

This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →