← ArchivePaper2018
SFV: Reinforcement Learning of Physical Skills from Video
Xue Bin Peng, Angjoo Kanazawa, Jitendra Malik, Pieter Abbeel, Sergey Levine
Physics-based character learns to imitate athletic motions reconstructed from monocular video using deep RL.
Abstract
Data-driven character animation based on motion capture can produce highly naturalistic behaviors and, when combined with physics simulation, can provide for natural procedural responses to physical perturbations, environmental changes, and morphological discrepancies. Motion capture remains the most popular source of motion data, but collecting mocap data typically requires heavily instrumented environments and actors. In this paper, we propose a method that enables physically simulated characters to learn skills from videos (SFV). Our approach, based on deep pose estimation and deep reinforcement learning, allows data-driven animation to leverage the abundance of publicly available video clips from the web, such as those from YouTube. This has the potential to enable fast and easy design of character controllers simply by querying for video recordings of the desired behavior. The resulting controllers are robust to perturbations, can be adapted to new settings, can perform basic object interactions, and can be retargeted to new morphologies via reinforcement learning. We further demonstrate that our method can predict potential human motions from still images, by forward simulation of learned controllers initialized from the observed pose. Our framework is able to learn a broad range of dynamic skills, including locomotion, acrobatics, and martial arts. (Video1)
How to read this
- Category
- Method: learning physics-based character skills from video (SFV)
- Contributions
- Enables physically simulated characters to learn skills from ordinary monocular video clips
- Combines deep pose estimation with deep reinforcement learning to imitate the reconstructed motion
- Produces controllers robust to perturbations, adaptable to new settings and object interactions, and retargetable to new morphologies; can also predict motion from still images
- Context
- Builds on example-guided physics-based RL (DeepMimic) and deep human pose estimation, replacing mocap reference with motion reconstructed from web video.Builds on: DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
- Correctness
- Demonstrated on athletic skills reconstructed from video and on perturbation robustness; a reader should keep in mind that quality is bounded by the monocular pose-estimation accuracy and by the RL reward and simulation, so noisy or ambiguous video limits the learned skill.
- Clarity
- Accessible high-level pipeline with technical depth in the RL stage; a first pass conveys the video-to-controller idea, a second pass covers pose-estimation and reward details.
- How to read it
- First pass for the video -> pose -> RL-imitation pipeline and what it unlocks; do a second pass on how pose-estimation noise is handled and how the reward relates to DeepMimic if you want to build on it.
Builds on
Built upon by
Nothing yet.
Related work
- DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills 2018 / SIGGRAPH
- Physics-based Motion Capture Imitation with Deep Reinforcement Learning 2018 / MIG
- C·ASE: Learning Conditional Adversarial Skill Embeddings for Physics-based Characters 2023 / SIGGRAPH Asia
- ControlVAE: Model-Based Learning of Generative Controllers for Physics-Based Characters 2022 / SIGGRAPH Asia
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →