← ArchivePaper2019
Learning Character-Agnostic Motion for Motion Retargeting in 2D
Kfir Aberman, Rundi Wu, Dani Lischinski, Baoquan Chen, Daniel Cohen-Or
Disentangled motion representation separating content from style enabling video-to-video motion retargeting without 3D reconstruction.
Abstract
Analyzing human motion is a challenging task with a wide variety of applications in computer vision and in graphics. One such application, of particular importance in computer animation, is the retargeting of motion from one performer to another. While humans move in three dimensions, the vast majority of human motions are captured using video, requiring 2D-to-3D pose and camera recovery, before existing retargeting approaches may be applied. In this paper, we present a new method for retargeting video-captured motion between different human performers, without the need to explicitly reconstruct 3D poses and/or camera parameters. In order to achieve our goal, we learn to extract, directly from a video, a high-level latent motion representation, which is invariant to the skeleton geometry and the camera view. Our key idea is to train a deep neural network to decompose temporal sequences of 2D poses into three components: motion, skeleton, and camera view-angle. Having extracted such a representation, we are able to re-combine motion with novel skeletons and camera views, and decode a retargeted temporal sequence, which we compare to a ground truth from a synthetic dataset. We demonstrate that our framework can be used to robustly extract human motion from videos, bypassing 3D reconstruction, and outperforming existing retargeting methods, when applied to videos in-the-wild.
How to read this
- Category
- Method: deep learning for 2D video motion retargeting
- Contributions
- Retargets video-captured motion between performers without explicit 3D pose or camera reconstruction
- Trains a network to decompose 2D pose sequences into three disentangled components: motion, skeleton, and camera view-angle
- Re-combines extracted motion with novel skeletons and camera views to decode a retargeted sequence, evaluated against synthetic ground truth
- Context
- Reframes the classic retargeting problem of Gleicher's 'Retargeting Motion to New Characters' (1998) in 2D video space via a learned, skeleton- and view-invariant latent motion representation.Builds on: Retargeting Motion to New Characters
- Correctness
- The core assumption is that 2D pose sequences cleanly factor into motion, skeleton, and view; quantitative evaluation uses synthetic data with known ground truth, so reported accuracy is most trustworthy in-distribution and real-video generalization should be read with caution.
- Clarity
- Well-motivated and accessible; a first pass conveys the three-way disentanglement, with the network architecture and training rewarding a second pass.
- How to read it
- First pass for the motion/skeleton/view decomposition idea and why it avoids 3D recovery; second pass on the architecture and synthetic-data evaluation to judge how far it transfers to real footage.
Builds on
Built upon by
Related work
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →