← ArchivePaper2022
QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatars
Deep RL drives a full-body simulated avatar in real time using only VR headset and controller signals as input.
Abstract
Real-time tracking of human body motion is crucial for interactive and immersive experiences in AR/VR. However, very limited sensor data about the body is available from standalone wearable devices such as HMDs (Head Mounted Devices) or AR glasses. In this work, we present a reinforcement learning framework that takes in sparse signals from an HMD and two controllers, and simulates plausible and physically valid full body motions. Using high quality full body motion as dense supervision during training, a simple policy network can learn to output appropriate torques for the character to balance, walk, and jog, while closely following the input signals. Our results demonstrate surprisingly similar leg motions to ground truth without any observations of the lower body, even when the input is only the 6D transformations of the HMD. We also show that a single policy can be robust to diverse locomotion styles, different body sizes, and novel environments.
How to read this
- Category
- Method: full-body motion tracking from sparse VR sensors
- Contributions
- A reinforcement learning framework that drives a physically valid full-body simulated avatar from only HMD and two controller signals in real time
- Use of high-quality full-body motion as dense supervision so a simple policy learns torques to balance, walk, and jog while following the input
- A single policy shown to be robust across diverse locomotion styles, body sizes, and novel environments, including plausible leg motion from upper-body input alone
- Context
- Builds on physics-based motion tracking with supervisory signals such as SuperTrack (Fussell et al. 2021), specialized to the sparse three-point VR input case.Builds on: SuperTrack: Motion Tracking for Physically Simulated Characters Using Supervisory Signals
- Correctness
- Lower-body motion is inferred without lower-body observations, so plausible leg motion is a learned prior conditioned on sparse upper-body signals rather than a measurement; accuracy on motions far from the training distribution is not guaranteed.
- Clarity
- Accessible; a first pass conveys the sparse-input-to-full-body idea, a second pass covers the RL training and supervision.
- How to read it
- Focus on what signals are available versus inferred and how dense supervision is used; second pass on the policy and reward design if you build sparse-input avatars.
Builds on
Built upon by
Related work
- UniCon: Universal Neural Controller for Physics-Based Character Motion 2020 / arXiv
- Physics-Based Motion Retargeting from Sparse Inputs 2023 / SCA
- DReCon: Data-Driven Responsive Control of Physics-Based Characters 2019 / TOG
- SuperTrack: Motion Tracking for Physically Simulated Characters Using Supervisory Signals 2021 / SIGGRAPH Asia
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →