← ArchivePaper2024

MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representations

Heyuan Yao, Zhenhua Song, Yuyang Zhou, Tenglong Ao, Baoquan Chen, Libin Liu

VQ-VAE with model-based RL learns scalable discrete motion codes enabling universal tracking, interactive control, and GPT-based motion generation.

Open the paper ↗Trace in the lineage graph

Abstract

In this work, we present MoConVQ, a novel unified framework for physics-based motion control leveraging scalable discrete representations. Building upon vector quantized variational autoencoders (VQ-VAE) and model-based reinforcement learning, our approach effectively learns motion embeddings from a large, unstructured dataset spanning tens of hours of motion examples. The resultant motion representation not only captures diverse motion skills but also offers a robust and intuitive interface for various applications. We demonstrate the versatility of MoConVQ through several applications: universal tracking control from various motion sources, interactive character control with latent motion representations using supervised learning, physics-based motion generation from natural language descriptions using the GPT framework, and, most interestingly, seamless integration with large language models (LLMs) with in-context learning to tackle complex and abstract tasks.

How to read this

Category: Method: a discrete motion representation for physics-based control
Contributions: MoConVQ, a unified physics-based control framework using VQ-VAE motion embeddings learned via model-based RL
Learns motion codes from a large unstructured dataset spanning tens of hours of examples
Enables universal tracking, latent interactive control, GPT-based motion generation, and LLM integration
Context: Builds on model-based learning of physics-based controllers (e.g. Yao et al.'s ControlVAE) by replacing continuous latents with scalable discrete VQ-VAE codes.Builds on: ControlVAE: Model-Based Learning of Generative Controllers for Physics-Based Characters
Correctness: Assumes a discrete code space can capture diverse skills while staying controllable; versatility is shown across several applications, so a reader should gauge how robust each application is rather than treating breadth as uniform quality.
Clarity: Moderately technical; a first pass conveys the discrete-representation idea, a second pass for the VQ-VAE plus model-based RL formulation.
How to read it: Focus on the discrete representation and how it interfaces with tracking, RL, and the GPT/LLM layers; a second and possibly third pass help if you care about the training detail.

Builds on

ControlVAE: Model-Based Learning of Generative Controllers for Physics-Based Characters 2022

Built upon by

Nothing yet.

Related work

MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpainting 2024 / SIGGRAPH Asia
TEMOS: Generating Diverse Human Motions from Textual Descriptions 2022 / CVPR
PDP: Physics-Based Character Animation via Diffusion Policy 2024 / SIGGRAPH Asia
MuscleVAE: Model-Based Controllers of Muscle-Actuated Characters 2023 / SIGGRAPH Asia

Keywords

moconvq vq-vae vector quantized variational autoencoder residual vector quantization physics-based motion control model-based reinforcement learning world model discrete motion representation motion tracking gpt transformer large language model in-context learning text-to-motion codebook mixture-of-experts amass pd control

This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →