← ArchivePaper2023

Physics-Based Motion Retargeting from Sparse Inputs

Daniele Reda, Jungdam Won, Yuting Ye, Michiel van de Panne, Alexander Winkler

SCAAcademic25 citesRetargeting Motion Synthesis

RL policy retargets sparse VR sensor streams to morphologically diverse characters including non-human avatars in real time.

Open the paper ↗Trace in the lineage graph

Abstract

Avatars are important to create interactive and immersive experiences in virtual worlds. One challenge in animating these characters to mimic a user's motion is that commercial AR/VR products consist only of a headset and controllers, providing very limited sensor data of the user's pose. Another challenge is that an avatar might have a different skeleton structure than a human and the mapping between them is unclear. In this work we address both of these challenges. We introduce a method to retarget motions in real-time from sparse human sensor data to characters of various morphologies. Our method uses reinforcement learning to train a policy to control characters in a physics simulator. We only require human motion capture data for training, without relying on artist-generated animations for each avatar. This allows us to use large motion capture datasets to train general policies that can track unseen users from real and sparse data in real-time. We demonstrate the feasibility of our approach on three characters with different skeleton structure: a dinosaur, a mouse-like creature and a human. We show that the avatar poses often match the user surprisingly well, despite having no sensor information of the lower body available.

How to read this

Category: Method: physics-based real-time motion retargeting from sparse inputs
Contributions: A method to retarget motion in real time from sparse AR/VR sensor data (headset plus controllers) to characters of varied morphology
An RL policy trained in a physics simulator using only human mocap data, with no per-avatar artist animation
Demonstration across non-human skeletons (dinosaur, mouse-like creature) and a human, tracking unseen users from sparse real data
Context: Combines contact-aware retargeting (Villegas 2021) with sparse-sensor simulated tracking such as QuestSim (Winkler 2022).Builds on: Contact-Aware Retargeting of Skinned Motion · QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatars
Correctness: Shown on three characters with differing skeletons in simulation; the human-to-nonhuman mapping is learned and acknowledged as not fully defined, so match quality varies and is reported as often (not always) good.
Clarity: Accessible in setup; a first pass conveys the sparse-input-to-avatar pipeline, a second pass clarifies the RL formulation and reward design.
How to read it: First pass for the problem framing (sparse sensors, diverse morphologies) and feasibility claims; second pass on the RL policy and physics setup if building VR avatar retargeting.

Builds on

Built upon by

Nothing yet.

Related work

DReCon: Data-Driven Responsive Control of Physics-Based Characters 2019 / TOG
CALM: Conditional Adversarial Latent Models for Directable Virtual Characters 2023 / SIGGRAPH
QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatars 2022 / SIGGRAPH Asia
Perpetual Humanoid Control for Real-time Simulated Avatars 2023 / ICCV

Keywords

motion retargeting reinforcement learning physics-based simulation sparse inputs proximal policy optimization kinematic retargeting imitation reward contact reward asymmetric actor-critic non-human characters head-mounted device motion capture avatars isaac gym locomotion morphology

This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →