ContactVision: Learning Foot Contact from Video for Physically Plausible Gait Animation

How to read this

Category

Academic Eurographics paper combining a video based perception model with a reinforcement learning animation system for gait

Contributions

• Trains a network to detect heel and toe contact states directly from video, using contact labels derived from motion capture via ground reaction force estimation, so it can be trained on existing mocap datasets without extra hardware
• Feeds the predicted contact labels into a reinforcement learning framework built around a two segment foot model, rather than a single rigid foot, for more realistic foot articulation
• Demonstrates the contact detection network in two downstream uses, gait motion reconstruction and gait analysis, showing the same perception output serves both an animation and a biomechanics use case

Context

This sits in the video based motion capture and physically simulated gait line of animation research, pairing a perception problem, detecting contact from video, with a control problem, using that contact signal inside a reinforcement learning foot controller. It has no builds_on entry in the archive, but it clearly follows physically plausible locomotion work that uses reinforcement learning with articulated foot models rather than kinematic retargeting.

Correctness

The method is trained with supervised labels derived from mocap based ground reaction force estimation, giving it a concrete, checkable training signal, and it is evaluated in two downstream tasks, reconstruction and analysis. This is a full Eurographics paper in Computer Graphics Forum, so expect a comparison against baseline contact estimation methods, check the qualitative comparison figures alongside the paper for how much better it looks against simpler baselines.

Clarity

Moderately technical, it spans two different subfields, a video perception network and a physics based reinforcement learning controller, so expect some background needed in both. The two segment foot model detail suggests it is written with animation and biomechanics readers in mind rather than pure ML.

How to read it

First pass, read the abstract and any contact detection accuracy figure to see how reliable the video based contact signal is. Second pass, read how the reinforcement learning framework consumes the contact labels and what the two segment foot model buys over a single rigid foot. Third pass is worth it for anyone building video driven mocap cleanup or a physically simulated locomotion controller who needs a contact estimation front end.