← ArchivePaper2022

TEMOS: Generating Diverse Human Motions from Textual Descriptions

Mathis Petrovich, Michael J. Black, Gül Varol

Variational transformer encodes text and motion into a joint latent space, generating diverse motion sequences from a single description.

Open the paper ↗Trace in the lineage graph

Abstract

We address the problem of generating diverse 3D human motions from textual descriptions. This challenging task requires joint modeling of both modalities: understanding and extracting useful human-centric information from the text, and then generating plausible and realistic sequences of human poses. In contrast to most previous work which focuses on generating a single, deterministic, motion from a textual description, we design a variational approach that can produce multiple diverse human motions. We propose TEMOS, a text-conditioned generative model leveraging variational autoencoder (VAE) training with human motion data, in combination with a text encoder that produces distribution parameters compatible with the VAE latent space. We show the TEMOS framework can produce both skeleton-based animations as in prior work, as well more expressive SMPL body motions. We evaluate our approach on the KIT Motion-Language benchmark and, despite being relatively straightforward, demonstrate significant improvements over the state of the art. Code and models are available on our webpage.

How to read this

Category: Method: text-conditioned human motion generation
Contributions: TEMOS, a variational (VAE) generative model that produces multiple diverse 3D human motions from a single textual description
A text encoder that outputs distribution parameters compatible with the motion VAE latent space, jointly modeling text and motion
Generation of both skeleton-based animations and more expressive SMPL body motions
Context: Extends transformer-VAE motion synthesis (Petrovich et al., ACTOR, 2021) from action-conditioned generation toward free-text conditioning with diverse, non-deterministic output.Builds on: Action-Conditioned 3D Human Motion Synthesis with Transformer VAE
Correctness: Evaluated on the KIT Motion-Language benchmark with reported improvements over prior work; as the authors note it is relatively straightforward, so generalization beyond that benchmark's vocabulary and motion distribution is the main caveat.
Clarity: Accessible to readers with VAE and transformer background; a first pass conveys the joint-latent-space idea, a second pass for the training objective and diversity mechanism.
How to read it: Read once for the text-and-motion shared-latent-space design; second pass on the VAE training and how the text encoder is aligned to the latent space, plus the KIT evaluation protocol.

Builds on

Action-Conditioned 3D Human Motion Synthesis with Transformer VAE 2021

Built upon by

Nothing yet.

Related work

Action-Conditioned 3D Human Motion Synthesis with Transformer VAE 2021 / ICCV
Executing Your Commands via Motion Diffusion in Latent Space 2023 / CVPR
Generating Diverse and Natural 3D Human Motions from Text 2022 / CVPR
MotionCLIP: Exposing Human Motion Generation to CLIP Space 2022 / CVPR

Keywords

text-to-motion variational autoencoder transformer smpl cross-modal latent space distilbert positional encoding text-conditioned motion generation kit motion-language amass sequence-level embedding kl divergence reparameterization trick skeleton representation diverse motion generation 6d rotation representation

This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →