← ArchivePaper2022

MotionCLIP: Exposing Human Motion Generation to CLIP Space

Guy Tevet, Brian Gordon, Amir Hertz, Amit H. Bermano, Daniel Cohen-Or

Aligns a 3D human motion auto-encoder latent space with CLIP, enabling out-of-domain text and image-driven motion synthesis.

Open the paper ↗Trace in the lineage graph

Abstract

We introduce MotionCLIP, a 3D human motion auto-encoder featuring a latent embedding that is disentangled, well behaved, and supports highly semantic textual descriptions. MotionCLIP gains its unique power by aligning its latent space with that of the Contrastive Language-Image Pre-training (CLIP) model. Aligning the human motion manifold to CLIP space implicitly infuses the extremely rich semantic knowledge of CLIP into the manifold. In particular, it helps continuity by placing semantically similar motions close to one another, and disentanglement, which is inherited from the CLIP-space structure. MotionCLIP comprises a transformer-based motion auto-encoder, trained to reconstruct motion while being aligned to its text label's position in CLIP-space. We further leverage CLIP's unique visual understanding and inject an even stronger signal through aligning motion to rendered frames in a self-supervised manner. We show that although CLIP has never seen the motion domain, MotionCLIP offers unprecedented text-to-motion abilities, allowing out-of-domain actions, disentangled editing, and abstract language specification. For example, the text prompt"couch"is decoded into a sitting down motion, due to lingual similarity, and the prompt"Spiderman"results in a web-swinging-like solution that is far from seen during training.

How to read this

Category: Method: text/image-driven human motion generation via CLIP alignment
Contributions: A transformer-based 3D human motion auto-encoder whose latent space is aligned with CLIP space, infusing CLIP's semantic structure into the motion manifold
Adds a self-supervised signal by aligning motion to rendered frames, leveraging CLIP's visual understanding
Enables out-of-domain text-to-motion, disentangled editing, and abstract language specification
Context: Builds on transformer VAE motion synthesis such as Petrovich et al. ACTOR (2021) and bridges it to CLIP's language-image embedding space.Builds on: Action-Conditioned 3D Human Motion Synthesis with Transformer VAE
Correctness: Shows that aligning to CLIP yields a well-behaved, disentangled latent enabling out-of-domain prompts; results inherit CLIP's semantic structure, so quality and biases are bounded by CLIP and by the motion training data, and abstract prompts give plausible rather than ground-truth motions.
Clarity: Conceptually elegant and accessible; a first pass conveys the alignment idea, a second pass covers the auto-encoder and the alignment losses.
How to read it: First pass for why aligning a motion manifold to CLIP buys semantic editing; second pass on the dual alignment (text label plus rendered-frame) losses if you want to reproduce or extend it.

Builds on

Action-Conditioned 3D Human Motion Synthesis with Transformer VAE 2021

Built upon by

Nothing yet.

Related work

Human Motion Diffusion Model 2022 / arXiv
TEMOS: Generating Diverse Human Motions from Textual Descriptions 2022 / CVPR
PDP: Physics-Based Character Animation via Diffusion Policy 2024 / SIGGRAPH Asia
CLoSD: Closing the Loop between Simulation and Diffusion for Multi-Task Character Control 2024 / arXiv

Keywords

motionclip human motion generation text-to-motion clip motion auto-encoder transformer encoder transformer decoder latent space alignment disentanglement motion style transfer smpl babel amass action recognition latent space editing motion interpolation 6d rotation representation

This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →