← ArchivePaper2024
Physical Based Motion Reconstruction From Videos Using Musculoskeletal Model
Hierarchical control with a muscle layer and trajectory layer reconstructs physically plausible character motion from monocular video.
Abstract
We propose a novel method that combines human pose estimation and physical simulation of character animation. Our approach allows characters to learn from the actor's skills captured in videos and subsequently reconstruct the motions with high fidelity in a physically simulated environment. Firstly, we model the character based on the human musculoskeletal system and build a complete dynamics model of the proposed system using the Lagrange equations of motion. Next, we employ the pose estimation method to process the input video and generate human reference motion. Finally, we design a hierarchical control framework consisting of a trajectory tracking layer and a muscle control layer. The trajectory tracking layer aims to minimize the difference between the reference motion pose and the actual output pose, while the muscle control layer aims to minimize the difference between the target torque and the actual output muscle force. The two layers interact by passing parameters through a proportional differential controller until the desired learning objective is achieved. A series of complex experimental results demonstrate that our proposed method can learn to produce comparable high‐quality motions with high similarity from videos of different complexity levels and remains stable in the presence of muscle contracture weakness perturbations.
How to read this
- Category
- Method: physics-based motion reconstruction from video
- Contributions
- A method combining pose estimation with physics simulation to reconstruct character motion from monocular video
- A musculoskeletal character with full dynamics built from the Lagrange equations of motion
- A hierarchical controller with a trajectory-tracking layer and a muscle-control layer coupled via a PD controller
- Context
- Joins video-based pose estimation with muscle-actuated simulation, relating to scalable muscle-actuated control such as Lee et al.'s scalable muscle-actuated human simulation.Builds on: Scalable Muscle-Actuated Human Simulation and Control
- Correctness
- Quality is bounded by the monocular pose-estimation reference and the fidelity of the musculoskeletal dynamics model; results are reported on complex experiments, so a reader should note dependence on the input pose estimate and the two-layer convergence behavior.
- Clarity
- Moderately technical; a first pass conveys the pipeline and two-layer control, a second pass for the Lagrangian dynamics and the layer interaction.
- How to read it
- Read the hierarchical control framework and how the trajectory and muscle layers exchange parameters; a second pass pays off for the dynamics derivation and reconstruction fidelity.
Builds on
Built upon by
Nothing yet.
Related work
- Physics-Based Character Controllers Using Conditional VAEs 2022 / SIGGRAPH
- DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills 2018 / SIGGRAPH
- Generative GaitNet 2022 / SIGGRAPH
- Scalable Muscle-Actuated Human Simulation and Control 2019 / SIGGRAPH
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →