← ArchivePaper2023
ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model
Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, Ziwei Liu
Integrates hybrid semantic-kinematic retrieval into a diffusion model to improve rare-motion generation quality.
Abstract
3D human motion generation is crucial for creative industry. Recent advances rely on generative models with domain knowledge for text-driven motion generation, leading to substantial progress in capturing common motions. However, the performance on more diverse motions remains unsatisfactory. In this work, we propose ReMoDiffuse, a diffusion-model-based motion generation framework that integrates a retrieval mechanism to refine the denoising process. ReMoDiffuse enhances the generalizability and diversity of text-driven motion generation with three key designs: 1) Hybrid Retrieval finds appropriate references from the database in terms of both semantic and kinematic similarities. 2) Semantic-Modulated Transformer selectively absorbs retrieval knowledge, adapting to the difference between retrieved samples and the target motion sequence. 3) Condition Mixture better utilizes the retrieval database during inference, overcoming the scale sensitivity in classifier-free guidance. Extensive experiments demonstrate that ReMoDiffuse outperforms state-of-the-art methods by balancing both text-motion consistency and motion quality, especially for more diverse motion generation. Project page: https://mingyuan-zhang.github.io/projects/ReMoDiffuse.html
How to read this
- Category
- Method: a retrieval-augmented motion diffusion model for text-driven generation
- Contributions
- Hybrid Retrieval that selects database references by both semantic and kinematic similarity
- Semantic-Modulated Transformer that selectively absorbs retrieved knowledge to adapt to the target sequence
- Condition Mixture to better use the retrieval database at inference and reduce scale sensitivity in classifier-free guidance
- Context
- Builds on text-to-motion diffusion in the lineage of Human Motion Diffusion Model (Tevet et al.), adding a retrieval mechanism to refine denoising.Builds on: Human Motion Diffusion Model
- Correctness
- The work argues improved diversity and rare-motion quality from retrieval, and reports outperforming prior methods on text-motion consistency and quality, but gains depend on the coverage and quality of the retrieval database, so out-of-database motions remain a fair concern.
- Clarity
- Accessible at a high level; a first pass conveys the retrieval idea, a second pass is needed for the transformer modulation and condition-mixture formulation.
- How to read it
- Focus on how retrieval is integrated into the denoising loop and why it helps rare motions; do a second pass on the Semantic-Modulated Transformer and Condition Mixture if you intend to reproduce or extend it.
Builds on
Built upon by
Nothing yet.
Related work
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →