← ArchivePaper2022

PADL: Language-Directed Physics-Based Character Control

Jordan Juravsky, Yunrong Guo, Sanja Fidler, Xue Bin Peng

SIGGRAPH AsiaAcademic67 cites1 descendantMotion Synthesis

Combines NLP with physics-based character control so natural language commands specify tasks and low-level motion skills.

Open the paper ↗Trace in the lineage graph

Abstract

Developing systems that can synthesize natural and life-like motions for simulated characters has long been a focus for computer animation. But in order for these systems to be useful for downstream applications, they need not only produce high-quality motions, but must also provide an accessible and versatile interface through which users can direct a character’s behaviors. Natural language provides a simple-to-use and expressive medium for specifying a user’s intent. Recent breakthroughs in natural language processing (NLP) have demonstrated effective use of language-based interfaces for applications such as image generation and program synthesis. In this work, we present PADL, which leverages recent innovations in NLP in order to take steps towards developing language-directed controllers for physics-based character animation. PADL allows users to issue natural language commands for specifying both high-level tasks and low-level skills that a character should perform. We present an adversarial imitation learning approach for training policies to map high-level language commands to low-level controls that enable a character to perform the desired task and skill specified by a user’s commands.

How to read this

Category: Method: language-directed physics-based character control
Contributions: PADL, a system that lets users issue natural-language commands specifying both high-level tasks and low-level skills for a simulated character
An adversarial imitation learning approach that maps language commands to low-level controls
Steps toward an accessible language-based interface for physics-based animation, drawing on recent NLP advances
Context: Combines NLP language interfaces with adversarial skill embeddings for physically simulated characters in the lineage of ASE (Peng et al. 2022).Builds on: ASE: Large-Scale Reusable Adversarial Skill Embeddings for Physically Simulated Characters
Correctness: Validated within simulated physics-based control, so behavior coverage is bounded by the imitation data and the skills the policy learned, and language grounding is only as broad as the trained command set.
Clarity: Accessible framing; a first pass conveys the language-to-control idea, a second pass is needed for the adversarial imitation training.
How to read it: Read first for how language maps to high-level tasks versus low-level skills; second pass on the imitation-learning objective if you want to reproduce or extend the controller.

Builds on

ASE: Large-Scale Reusable Adversarial Skill Embeddings for Physically Simulated Characters 2022

Built upon by

SuperPADL: Scaling Language-Directed Physics-Based Control with Progressive Supervised Distillation 2024

Related work

SuperPADL: Scaling Language-Directed Physics-Based Control with Progressive Supervised Distillation 2024 / SIGGRAPH
CALM: Conditional Adversarial Latent Models for Directable Virtual Characters 2023 / SIGGRAPH
AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control 2021 / SIGGRAPH
UniCon: Universal Neural Controller for Physics-Based Character Motion 2020 / arXiv

Keywords

padl language-directed control physics-based character animation adversarial imitation learning reinforcement learning skill embedding clip text encoder transformer autoencoder motionclip multi-task aggregation multiple-choice question answering bert proximal policy optimization isaac gym skill interpolation adversarial discriminator natural language commands humanoid character

This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →