← ArchivePaper2023
T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations
Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen
VQ-VAE plus GPT framework for text-to-motion that outperforms diffusion baselines with FID 0.116 on HumanML3D.
Abstract
In this work, we investigate a simple and must-known conditional generative framework based on Vector Quantised-Variational AutoEncoder (VQ-VAE) and Generative Pre-trained Transformer (GPT) for human motion generation from textural descriptions. We show that a simple CNN-based VQ-VAE with commonly used training recipes (EMA and Code Reset) allows us to obtain high-quality discrete representations. For GPT, we incorporate a simple corruption strategy during the training to alleviate training-testing discrepancy. Despite its simplicity, our T2M-GPT shows better performance than competitive approaches, including recent diffusion-based approaches. For example, on HumanML3D, which is currently the largest dataset, we achieve comparable performance on the consistency between text and generated motion (R-Precision), but with FID 0.116 largely outperforming MotionDiffuse of 0.630. Additionally, we conduct analyses on HumanML3D and observe that the dataset size is a limitation of our approach. Our work suggests that VQ-VAE still remains a competitive approach for human motion generation. Our implementation is available on the project page: https://mael-zys.github.io/T2M-GPT/.
How to read this
- Category
- Method: text-to-motion generation (VQ-VAE plus GPT)
- Contributions
- A two-stage framework: a CNN-based VQ-VAE that learns discrete motion tokens, then a GPT that generates token sequences from text.
- Training recipes (EMA and Code Reset) for high-quality codebooks, plus a corruption strategy to reduce the train-test gap.
- Reports strong FID on HumanML3D, presented as competitive with or better than diffusion-based approaches.
- Context
- Builds on the HumanML3D text-to-motion dataset (Guo et al., 2022) and adapts the VQ-VAE plus autoregressive transformer recipe to human motion as an alternative to diffusion baselines like MotionDiffuse.Builds on: Generating Diverse and Natural 3D Human Motions from Text
- Correctness
- Validated quantitatively on HumanML3D using FID and R-Precision; the authors themselves flag dataset size as a limiting factor, so generalization claims should be read with that caveat.
- Clarity
- Accessible framing (deliberately 'simple'); a first pass conveys the architecture, a second pass is needed for the VQ-VAE training details and corruption strategy.
- How to read it
- Focus on Section/figures for the two-stage pipeline and the metrics table; do a second pass on the codebook recipes (EMA, Code Reset) and corruption trick if you plan to reproduce or compare against diffusion methods.
Built upon by
Nothing yet.
Related work
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →