← ArchivePaper2019

Multi-Objective Adversarial Gesture Generation

Ylva Ferstl, Michael Neff, Rachel McDonnell

MIGAcademic93 cites1 descendantMotion Synthesis Facial

GAN framework maps speech to 3D conversational gesture motion using separate adversaries for dynamics, joint plausibility, and diversity.

Open the paper ↗Trace in the lineage graph

Abstract

Applications for conversational virtual agents are on the rise, but producing realistic non-verbal behavior for spoken utterances remains an unsolved problem. We explore the use of a generative adversarial training paradigm to map speech to 3D gesture motion. We define the gesture generation problem as a series of smaller sub-problems, including plausible gesture dynamics, realistic joint configurations, and diverse and smooth motion. Each sub-problem is monitored by separate adversaries. For the problem of enforcing realistic gesture dynamics in our output, we train a classifier to automatically detect gesture phases. We find adversarial training to be superior to the use of a standard regression loss and discuss the benefit of each of our training objectives. We recorded a dataset of over 6 hours of natural, unrehearsed speech with high-quality motion capture, as well as audio and video recording.

How to read this

Category: Method: a GAN-based speech-to-gesture motion generation framework
Contributions: Maps speech to 3D conversational gesture motion using a multi-objective adversarial training paradigm
Decomposes generation into sub-problems (gesture dynamics, joint plausibility, diversity) each monitored by a separate adversary, including a learned gesture-phase classifier
Records over 6 hours of natural unrehearsed speech with high-quality motion capture plus audio and video
Context: Relates to data-driven non-verbal behavior synthesis for conversational virtual agents, framing speech-to-gesture as an adversarial (GAN) alternative to standard regression losses.
Correctness: Authors report adversarial training as superior to a regression baseline and discuss each objective's benefit; assessment of gesture realism is inherently subjective and tied to their own captured corpus, so claims should be read as comparative within that setup.
Clarity: Reasonably accessible if familiar with GANs; a first pass conveys the multi-adversary decomposition, a second pass is needed for the per-adversary losses and phase classifier.
How to read it: First pass for the problem decomposition and why multiple adversaries; do a second pass on the adversary definitions and ablation discussion if you care about why adversarial beats regression here.

Builds on

Nothing in the archive, this is a starting point.

Built upon by

ZeroEGGS: Zero-Shot Example-Based Gesture Generation from Speech 2022

Related work

Robust Motion In-Betweening 2020 / SIGGRAPH
ZeroEGGS: Zero-Shot Example-Based Gesture Generation from Speech 2022 / Eurographics
MoGlow: Probabilistic and Controllable Motion Synthesis Using Normalising Flows 2020 / SIGGRAPH Asia
C·ASE: Learning Conditional Adversarial Skill Embeddings for Physics-based Characters 2023 / SIGGRAPH Asia

Keywords

gesture generation generative adversarial networks multiple discriminators speech-to-gesture gesture phase classification recurrent neural network lstm gated recurrent unit co-speech gesture motion capture mfcc prosodic features phase structure discriminator minibatch discrimination mode collapse conversational agents gesture stroke displacement discriminator

This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →