← ArchivePaper2023
SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation
Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, Fei Wang
Generates 3DMM motion coefficients for head pose and expression from audio and modulates a 3D-aware face renderer for stylized talking head generation from a single image.
Abstract
Generating talking head videos through a face image and a piece of speech audio still contains many challenges. i.e., unnatural head movement, distorted expression, and identity modification. We argue that these issues are mainly caused by learning from the coupled 2D motion fields. On the other hand, explicitly using 3D information also suffers problems of stiff expression and incoherent video. We present SadTalker, which generates 3D motion coefficients (head pose, expression) of the 3DMM from audio and implicitly modulates a novel 3D-aware face render for talking head generation. To learn the realistic motion coefficients, we explicitly model the connections between audio and different types of motion coefficients individually. Precisely, we present ExpNet to learn the accurate facial expression from audio by distilling both coefficients and 3D-rendered faces. As for the head pose, we design PoseVAE via a conditional VAE to synthesize head motion in different styles. Finally, the generated 3D motion coefficients are mapped to the unsupervised 3D keypoints space of the proposed face render to synthesize the final video. We conducted extensive experiments to demonstrate the superiority of our method in terms of motion and video quality.11The code and demo videos are available at https://sadtalker.github.io.
How to read this
- Category
- Method: audio-driven talking-head animation from a single image
- Contributions
- Generates 3DMM motion coefficients (head pose and expression) from audio and uses them to drive a 3D-aware face renderer
- ExpNet learns accurate expression coefficients from audio by distilling coefficients and 3D-rendered faces
- PoseVAE, a conditional VAE, synthesizes stylized head motion, with coefficients mapped to unsupervised 3D keypoints for the final video
- Context
- Builds on audio-driven facial reenactment work such as Neural Voice Puppetry (Thies et al.), moving from coupled 2D motion fields toward explicit 3DMM-mediated motion.Builds on: Neural Voice Puppetry: Audio-driven Facial Reenactment
- Correctness
- The premise is that decoupling pose and expression through 3DMM coefficients reduces unnatural motion and identity drift; results rest on a 3DMM and a learned renderer, so identity fidelity and expression range remain bounded by those models.
- Clarity
- Reasonably accessible; a first pass conveys the audio-to-coefficient-to-render pipeline, a second pass is needed for ExpNet and PoseVAE specifics.
- How to read it
- Trace the path from audio to 3DMM coefficients to the rendered frame; do a second pass on ExpNet and PoseVAE if you care about how realism and head-motion style are achieved.
Builds on
Built upon by
Nothing yet.
Related work
- Capture, Learning, and Synthesis of 3D Speaking Styles 2019 / CVPR
- MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement 2021 / CVPR
- CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior 2023 / CVPR
- FaceFormer: Speech-Driven 3D Facial Animation with Transformers 2022 / CVPR
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →