← ArchivePaper2017

Learning Locomotion Skills Using DeepRL: Does the Choice of Action Space Matter?

Systematic study of action space design choices for deep reinforcement learning locomotion controllers for simulated characters.

Open the paper ↗Trace in the lineage graph

Abstract

This paper studies how the choice of action parameterization affects learning difficulty and performance when using deep reinforcement learning to control articulated figure locomotion. It compares four actuation models, torques, muscle activations for musculotendon units, target joint angles for PD controllers, and target joint velocities, on a gait-cycle imitation task for planar bipeds, a dog, and a raptor. The policies are trained with an actor-critic method using positive temporal difference updates and experience replay, and MTU actuator parameters are tuned by alternating policy learning with covariance matrix adaptation. The results show that action spaces incorporating local feedback, such as PD target angles, improve learning speed, robustness, and motion quality, with the advantage growing as character complexity increases.

How to read this

Category: Empirical study: action-space design for DeepRL locomotion
Contributions: Systematic comparison of four actuation models (torques, muscle activations, PD target angles, target joint velocities) on a gait-imitation task
Finding that action spaces with local feedback, such as PD target angles, improve learning speed, robustness, and motion quality
Evidence that this advantage grows with character complexity, across planar bipeds, a dog, and a raptor
Context: Continues the authors' deep-RL locomotion line following Terrain-Adaptive Locomotion Skills Using Deep Reinforcement Learning, isolating the action-parameterization variable.Builds on: Terrain-Adaptive Locomotion Skills Using Deep Reinforcement Learning
Correctness: Conclusions are drawn from gait-cycle imitation on planar (2D) characters with an actor-critic method and specific tuning; readers should treat the PD-advantage finding as supported within that experimental setup and not assume it transfers unchanged to 3D or non-imitation tasks.
Clarity: Very readable as a controlled study; a first pass gives the takeaway, a second pass covers the training and MTU-tuning details.
How to read it: Read it for the practical conclusion on action representation; skim the RL machinery and focus on the comparison setup if you are choosing an action space for your own controller.

Builds on

Terrain-Adaptive Locomotion Skills Using Deep Reinforcement Learning 2016

Built upon by

Nothing yet.

Related work

Terrain-Adaptive Locomotion Skills Using Deep Reinforcement Learning 2016 / SIGGRAPH
DReCon: Data-Driven Responsive Control of Physics-Based Characters 2019 / TOG
Physical Based Motion Reconstruction From Videos Using Musculoskeletal Model 2024 / CASA
DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills 2018 / SIGGRAPH

Keywords

deep reinforcement learning action space actuation model joint torques muscle activations musculotendon units proportional-derivative controller target joint angles target joint velocities locomotion motion imitation actor-critic policy gradient positive temporal difference experience replay gait cycle articulated figure covariance matrix adaptation

This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →