← ArchivePaper2021
Semi-Supervised Video-Driven Facial Animation Transfer for Production
Lucio Moser, Chinyu Chien, Mark Williams, Jose Serra, Darren Hendler, Douglas Roble
Unsupervised image-to-image translation learns a shared latent space from video, then a supervised linear mapping drives character facial animation coefficients for production.
Abstract
We propose a simple algorithm for automatic transfer of facial expressions, from videos to a 3D character, as well as between distinct 3D characters through their rendered animations. Our method begins by learning a common, semantically-consistent latent representation for the different input image domains using an unsupervised image-to-image translation model. It subsequently learns, in a supervised manner, a linear mapping from the character images' encoded representation to the animation coefficients. At inference time, given the source domain (i.e., actor footage), it regresses the corresponding animation coefficients for the target character. Expressions are automatically remapped between the source and target identities despite differences in physiognomy. We show how our technique can be used in the context of markerless motion capture with controlled lighting conditions, for one actor and for multiple actors. Additionally, we show how it can be used to automatically transfer facial animation between distinct characters without consistent mesh parameterization and without engineered geometric priors. We compare our method with standard approaches used in production and with recent state-of-the-art models on single camera face tracking.
How to read this
- Category
- Method: production facial animation transfer (video-driven, semi-supervised)
- Contributions
- Learns a common semantically consistent latent space across image domains using an unsupervised image-to-image translation model
- Learns a supervised linear mapping from encoded character images to animation coefficients, regressing them from actor footage at inference
- Remaps expressions across differing physiognomies and transfers facial animation between characters without consistent mesh parameterization or engineered geometric priors
- Context
- Extends the authors' production facial-capture work (Masquerade, Moser et al.), shifting from fine-scale detail recovery toward cross-domain expression transfer via a shared learned latent space.Builds on: Masquerade: Fine-Scale Details for Head-Mounted Camera Motion Capture Data
- Correctness
- Demonstrated for markerless capture under controlled lighting for single and multiple actors and compared against standard production approaches; the controlled-lighting setting and the reliance on a learned shared latent space are practical assumptions a reader should note before generalizing to in-the-wild footage.
- Clarity
- Accessible given the simple two-stage (unsupervised latent plus supervised linear) recipe; a first pass conveys the pipeline, a second pass clarifies the translation model and coefficient regression.
- How to read it
- First pass for the latent-space-then-linear-map recipe and its production framing; do a second pass on the image-to-image model and the supervised mapping if you work on retargeting capture to characters.
Built upon by
Nothing yet.
Related work
- A Facial Motion Retargeting Pipeline for Appearance Agnostic 3D Characters 2024 / CASA
- Dog Code: Human to Quadruped Embodiment Using Shared Codebooks 2024 / MIG
- Transferring Facial Expressions to Different Face Models 2006 / SIACG
- Facial Retargeting with Automatic Range of Motion Alignment 2017 / SIGGRAPH
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →