← ArchivePaper2022
ZeroEGGS: Zero-Shot Example-Based Gesture Generation from Speech
Saeed Ghorbani, Ylva Ferstl, Daniel Holden, Nikolaus F. Troje, Marc-Andre Carbonneau
Zero-shot speech-driven full-body gesture generation using example style references, trained on a diverse high-quality motion dataset.
Abstract
We present ZeroEGGS, a neural network framework for speech‐driven gesture generation with zero‐shot style control by example. This means style can be controlled via only a short example motion clip, even for motion styles unseen during training. Our model uses a Variational framework to learn a style embedding, making it easy to modify style through latent space manipulation or blending and scaling of style embeddings. The probabilistic nature of our framework further enables the generation of a variety of outputs given the input, addressing the stochastic nature of gesture motion. In a series of experiments, we first demonstrate the flexibility and generalizability of our model to new speakers and styles. In a user study, we then show that our model outperforms previous state‐of‐the‐art techniques in naturalness of motion, appropriateness for speech, and style portrayal. Finally, we release a high‐quality dataset of full‐body gesture motion including fingers, with speech, spanning across 19 different styles. Our code and data are publicly available at https://github.com/ubisoft/ubisoft‐laforge‐ZeroEGGS.
How to read this
- Category
- Method: a speech-driven gesture generation model
- Contributions
- Zero-shot style control of full-body gesture via a single short example motion clip, including styles unseen in training
- A variational style embedding allowing latent manipulation, blending, and scaling, with probabilistic sampling for varied outputs
- Release of a high-quality full-body (with fingers) gesture-plus-speech dataset spanning 19 styles
- Context
- Builds on prior speech-to-gesture synthesis such as Ferstl's Multi-Objective Adversarial Gesture Generation, replacing adversarial training with a variational example-based style framework.Builds on: Multi-Objective Adversarial Gesture Generation
- Correctness
- Naturalness, appropriateness for speech, and style portrayal are assessed via a user study claiming to outperform prior state of the art, so claims rest on perceptual ratings; readers should remember user-study results are subjective and tied to this dataset.
- Clarity
- Accessible; a first pass conveys the example-based style idea, do a second pass for the variational formulation and latent operations.
- How to read it
- Focus first on how the style example is encoded into the latent embedding and mixed with speech; do a second pass on the variational loss if you intend to reproduce or extend the style control.
Builds on
Built upon by
Nothing yet.
Related work
- Multi-Objective Adversarial Gesture Generation 2019 / MIG
- FaceFormer: Speech-Driven 3D Facial Animation with Transformers 2022 / CVPR
- Audio-Driven Facial Animation by Joint End-to-End Learning of Pose and Emotion 2017 / SIGGRAPH
- MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement 2021 / CVPR
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →