← ArchivePaper2023

Executing Your Commands via Motion Diffusion in Latent Space

Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, Gang Yu

Motion Latent Diffusion (MLD) compresses motion into a VAE latent space before diffusion, achieving two-orders-of-magnitude speedup over raw-sequence diffusion.

Open the paper ↗Trace in the lineage graph

Abstract

We study a challenging task, conditional human motion generation, which produces plausible human motion sequences according to various conditional inputs, such as action classes or textual descriptors. Since human motions are highly diverse and have a property of quite different distribution from conditional modalities, such as textual descriptors in natural languages, it is hard to learn a probabilistic mapping from the desired conditional modality to the human motion sequences. Besides, the raw motion data from the motion capture system might be redundant in sequences and contain noises; directly modeling the joint distribution over the raw motion sequences and conditional modalities would need a heavy computational over-head and might result in artifacts introduced by the captured noises. To learn a better representation of the various human motion sequences, we first design a powerful Variational AutoEncoder (VAE) and arrive at a representative and low-dimensional latent code for a human motion sequence. Then, instead of using a diffusion model to establish the connections between the raw motion sequences and the conditional inputs, we perform a diffusion process on the motion latent space.

How to read this

Category: Method: conditional human motion generation (latent-space diffusion)
Contributions: Motion Latent Diffusion (MLD), running the diffusion process in a learned motion latent space rather than on raw sequences
A VAE that compresses motion into a representative low-dimensional latent code
Large reported speedup over raw-sequence diffusion while supporting text- and action-conditioned generation
Context: Extends the motion-diffusion line from the Human Motion Diffusion Model (tevet-mdm-2022) by moving diffusion into a VAE latent space, echoing the latent-diffusion idea from image generation.Builds on: Human Motion Diffusion Model
Correctness: Assumes a well-behaved VAE latent captures motion adequately; the efficiency gain and denoising rely on that compression, so latent quality bounds fidelity and the speedup claims should be read against the conditioning tasks evaluated.
Clarity: Accessible if you know latent diffusion; a first pass conveys the compress-then-diffuse idea, a second pass clarifies the VAE design and conditioning.
How to read it: Focus on why diffusing in latent space helps (cost and noise reduction) and on the VAE; a second pass pays off for the conditioning mechanism and the efficiency comparison.

Builds on

Human Motion Diffusion Model 2022

Built upon by

Nothing yet.

Related work

Human Motion Diffusion Model 2022 / arXiv
Generating Diverse and Natural 3D Human Motions from Text 2022 / CVPR
TEMOS: Generating Diverse Human Motions from Textual Descriptions 2022 / CVPR
CLoSD: Closing the Loop between Simulation and Diffusion for Multi-Task Character Control 2024 / arXiv

Keywords

motion latent diffusion conditional human motion generation text-to-motion action-to-motion variational autoencoder diffusion model latent space transformer autoencoder long skip connections classifier-free guidance clip text encoder denoiser amass humanml3d kit motion-language smpl frechet inception distance ddim

This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →