← ArchivePaper2020
Unpaired Motion Style Transfer from Video to Animation
Kfir Aberman, Yijia Weng, Dani Lischinski, Daniel Cohen-Or, Baoquan Chen
Encodes motion into disentangled content and style latent codes, applying temporally invariant AdaIN to transfer style extracted directly from RGB video onto 3D animation.
Abstract
Transferring the motion style from one animation clip to another, while preserving the motion content of the latter, has been a long-standing problem in character animation. Most existing data-driven approaches are supervised and rely on paired data, where motions with the same content are performed in different styles. In addition, these approaches are limited to transfer of styles that were seen during training. In this paper, we present a novel data-driven framework for motion style transfer, which learns from an unpaired collection of motions with style labels, and enables transferring motion styles not observed during training. Furthermore, our framework is able to extract motion styles directly from videos, bypassing 3D reconstruction, and apply them to the 3D input motion. Our style transfer network encodes motions into two latent codes, for content and for style, each of which plays a different role in the decoding (synthesis) process. While the content code is decoded into the output motion by several temporal convolutional layers, the style code modifies deep features via temporally invariant adaptive instance normalization (AdaIN). Moreover, while the content code is encoded from 3D joint rotations, we learn a common embedding for style from either 3D or 2D joint positions, enabling style extraction from videos.
How to read this
- Category
- Method: unpaired motion style transfer from video to 3D animation
- Contributions
- A data-driven framework that learns motion style transfer from an unpaired collection of style-labeled motions, with no paired same-content data required
- The ability to extract motion style directly from RGB video, bypassing 3D reconstruction, and apply it to 3D input motion
- An encoder that splits motion into content and style codes, decoding content via temporal convolutions while style modulates features through temporally invariant AdaIN, including styles unseen in training
- Context
- Builds on the authors' skeleton-aware retargeting work and adapts the content/style + AdaIN disentanglement idea from image style transfer to character motion.Builds on: Skeleton-Aware Networks for Deep Motion Retargeting
- Correctness
- Demonstrated as unpaired, label-driven transfer that can generalize to unseen styles and to styles read from video; a reader should keep in mind that style quality from video depends on the underlying motion extraction and that disentanglement is learned rather than guaranteed.
- Clarity
- Accessible if you know AdaIN-style transfer; a first pass conveys the content/style split, a second pass covers the temporally invariant AdaIN mechanism.
- How to read it
- First pass on what 'content' vs 'style' mean here and the video-to-animation pathway; second pass on the temporally invariant AdaIN and unpaired training setup if you intend to reproduce it.
Builds on
Built upon by
Related work
- Adult2Child: Motion Style Transfer Using CycleGANs 2020 / MIG
- How to Train Your Dog: Neural Enhancement of Quadruped Animations 2021 / MIG
- Robust Motion In-Betweening 2020 / SIGGRAPH
- SketchiMo: Sketch-based Motion Editing for Articulated Characters 2016 / SIGGRAPH
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →