← ArchivePaper2024

CLoSD: Closing the Loop between Simulation and Diffusion for Multi-Task Character Control

Guy Tevet, Sigal Raab, Setareh Cohan, Daniele Reda, Zhengyi Luo, Xue Bin Peng, Amit H. Bermano, Michiel van de Panne

Autoregressive diffusion planner runs in closed-loop with an RL tracking controller enabling text-guided multi-task physics character animation.

Open the paper ↗Trace in the lineage graph

Abstract

Motion diffusion models and Reinforcement Learning (RL) based control for physics-based simulations have complementary strengths for human motion generation. The former is capable of generating a wide variety of motions, adhering to intuitive control such as text, while the latter offers physically plausible motion and direct interaction with the environment. In this work, we present a method that combines their respective strengths. CLoSD is a text-driven RL physics-based controller, guided by diffusion generation for various tasks. Our key insight is that motion diffusion can serve as an on-the-fly universal planner for a robust RL controller. To this end, CLoSD maintains a closed-loop interaction between two modules -- a Diffusion Planner (DiP), and a tracking controller. DiP is a fast-responding autoregressive diffusion model, controlled by textual prompts and target locations, and the controller is a simple and robust motion imitator that continuously receives motion plans from DiP and provides feedback from the environment. CLoSD is capable of seamlessly performing a sequence of different tasks, including navigation to a goal location, striking an object with a hand or foot as specified in a text prompt, sitting down, and getting up. https://guytevet.github.io/CLoSD-page/

How to read this

Category: Method: text-driven physics-based character control coupling diffusion and RL
Contributions: CLoSD, a text-driven RL physics-based controller guided on-the-fly by motion diffusion
A closed-loop interaction between a fast autoregressive Diffusion Planner (DiP) and a robust RL tracking/imitation controller
Seamless execution of a sequence of multi-task behaviors such as navigation, striking objects, and other goal-directed actions
Context: Unifies motion diffusion models and RL physics control, building on text-to-motion diffusion (Tevet et al.'s Human Motion Diffusion Model) with a tracking-imitation controller.Builds on: Human Motion Diffusion Model
Correctness: Key insight is that diffusion can act as an on-the-fly universal planner for a robust imitator; demonstrated on simulated physics tasks, so plausibility comes from simulation and the controller's tracking fidelity rather than real-world deployment, and the abstract is truncated on full quantitative limits.
Clarity: Accessible if familiar with diffusion and RL control; a first pass conveys the closed-loop architecture, a second pass for DiP's autoregression and the controller's training.
How to read it: Focus on the planner-controller loop and why closed-loop coupling beats either module alone; a second pass pays off for the autoregressive diffusion conditioning and imitation reward design.

Builds on

Human Motion Diffusion Model 2022

Built upon by

Nothing yet.

Related work

Perpetual Humanoid Control for Real-time Simulated Avatars 2023 / ICCV
SuperPADL: Scaling Language-Directed Physics-Based Control with Progressive Supervised Distillation 2024 / SIGGRAPH
Generating Diverse and Natural 3D Human Motions from Text 2022 / CVPR
TEMOS: Generating Diverse Human Motions from Textual Descriptions 2022 / CVPR

Keywords

motion diffusion model diffusion planner reinforcement learning physics-based character control motion tracking controller closed-loop control autoregressive diffusion text-to-motion perpetual humanoid control amass humanml3d smpl proximal policy optimization human-object interaction distilbert transformer decoder isaac gym motion imitation

This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →