← ArchivePaper2017

VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera at over 30fps

Dushyant Mehta, Srinath Sridhar, Oleksandr Sotnychenko, Helge Rhodin, Mohammad Shafiei, Hans-Peter Seidel, Weipeng Xu, Dan Casas, Christian Theobalt

SIGGRAPHAcademic886 cites2 descendantsRetargeting

Real-time full-body 3D pose estimation from a single RGB camera at over 30fps enabling markerless mocap for interactive applications.

Open the paper ↗Trace in the lineage graph

Abstract

We present the first real-time method to capture the full global 3D skeletal pose of a human in a stable, temporally consistent manner using a single RGB camera. Our method combines a new convolutional neural network (CNN) based pose regressor with kinematic skeleton fitting. Our novel fully-convolutional pose formulation regresses 2D and 3D joint positions jointly in real time and does not require tightly cropped input frames. A real-time kinematic skeleton fitting method uses the CNN output to yield temporally stable 3D global pose reconstructions on the basis of a coherent kinematic skeleton. This makes our approach the first monocular RGB method usable in real-time applications such as 3D character control---thus far, the only monocular methods for such applications employed specialized RGB-D cameras. Our method's accuracy is quantitatively on par with the best offline 3D monocular RGB pose estimation methods. Our results are qualitatively comparable to, and sometimes better than, results from monocular RGB-D approaches, such as the Kinect. However, we show that our approach is more broadly applicable than RGB-D solutions, i.e., it works for outdoor scenes, community videos, and low quality commodity RGB cameras.

How to read this

Category: Method: real-time monocular 3D human pose estimation
Contributions: First real-time method to capture stable, temporally consistent full global 3D skeletal pose from a single RGB camera at over 30fps
A fully-convolutional CNN pose regressor that jointly predicts 2D and 3D joint positions without tightly cropped input
A real-time kinematic skeleton fitting step that yields temporally stable global pose suitable for 3D character control
Context: Sits in the lineage of CNN-based 2D and 3D human pose estimation combined with kinematic skeleton fitting, extending such methods to real-time monocular RGB use where prior interactive solutions relied on RGB-D cameras like Kinect.
Correctness: Reported as quantitatively on par with the best offline monocular RGB methods and qualitatively comparable to or better than RGB-D, with broader applicability (outdoor and community video); as a single-RGB approach it inherits depth and scale ambiguity, so absolute global accuracy should be read with that caveat.
Clarity: Clear system-level paper; a first pass conveys the two-stage architecture, a second pass is needed for the network and the fitting formulation.
How to read it: First pass for the regressor-plus-fitting split and the real-time claim; do a second pass on the CNN formulation and the kinematic fitting if implementing or comparing pose estimators.

Builds on

Nothing in the archive, this is a starting point.

Built upon by

Expressive Body Capture: 3D Hands, Face, and Body from a Single Image 2019

Related work

Normalized Euclidean Distance Matrices for Human Motion Retargeting 2017 / MIG
Robust Marker Trajectory Repair for MOCAP Using Kinematic Reference 2019 / MIG
Dog Code: Human to Quadruped Embodiment Using Shared Codebooks 2024 / MIG
Motion Retargeting for Crowd Simulation 2015 / DigiPro

Keywords

vnect 3d human pose estimation monocular rgb convolutional neural network resnet50 fully-convolutional 2d heatmaps 3d location-maps kinematic skeleton fitting inverse kinematics root-relative joint positions bounding box tracking temporal filtering 1 euro filter bone length maps real-time motion capture human3.6m mpi-inf-3dhp

This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →