← ArchivePaper2022
Facial Animation with Disentangled Identity and Motion using Transformers
Prashanth Chandran, Gaspard Zoss, Markus Gross, Paulo Gotardo, Derek Bradley
Transformer-based model disentangling facial identity and motion for retargeted animation of novel characters from a reference performance.
Abstract
We propose a 3D+time framework for modeling dynamic sequences of 3D facial shapes, representing realistic non‐rigid motion during a performance. Our work extends neural 3D morphable models by learning a motion manifold using a transformer architecture. More specifically, we derive a novel transformer‐based autoencoder that can model and synthesize 3D geometry sequences of arbitrary length. This transformer naturally determines frame‐to‐frame correlations required to represent the motion manifold, via the internal self‐attention mechanism. Furthermore, our method disentangles the constant facial identity from the time‐varying facial expressions in a performance, using two separate codes to represent neutral identity and the performance itself within separate latent subspaces. Thus, the model represents identity‐agnostic performances that can be paired with an arbitrary new identity code and fed through our new identity‐modulated performance decoder; the result is a sequence of 3D meshes for the performance with the desired identity and temporal length. We demonstrate how our disentangled motion model has natural applications in performance synthesis, performance retargeting, key‐frame interpolation and completion of missing data, performance denoising and retiming, and other potential applications that include full 3D body modeling.
How to read this
- Category
- Method: a Transformer model for 3D facial motion with disentangled identity and motion
- Contributions
- A Transformer-based autoencoder that models and synthesizes 3D facial geometry sequences of arbitrary length, learning a motion manifold via self-attention
- Disentangles constant facial identity from time-varying expression using two separate latent codes
- An identity-modulated performance decoder enabling performance synthesis, retargeting to new identities, and keyframing applications
- Context
- Extends neural 3D morphable models into the temporal domain, continuing the identity/expression-separation idea of Vlasic et al.'s Face Transfer with Multilinear Models using a Transformer.Builds on: Face Transfer with Multilinear Models
- Correctness
- Hinges on whether self-attention cleanly separates an identity-agnostic motion manifold from identity, and on retargeting being faithful when an arbitrary identity code is paired with a learned performance; demonstrated through synthesis and retargeting examples, so quality of disentanglement on unseen identities is the reader's main caveat.
- Clarity
- Accessible with Transformer/autoencoder background; a first pass conveys the disentanglement idea, a second pass for the latent-subspace and decoder formulation.
- How to read it
- First pass for the 3D+time disentanglement concept and its retargeting use cases; second pass on the autoencoder structure and identity-modulated decoder if you want to apply it to new characters.
Builds on
Built upon by
Nothing yet.
Related work
- Deep Appearance Models for Face Rendering 2018 / SIGGRAPH
- CANRIG: Cross-Attention Neural Face Rigging with Variable Local Control 2026 / Eurographics
- FaceBaker: Baking Character Facial Rigs with Machine Learning 2020 / SIGGRAPH
- Monocular Facial Performance Capture via Deep Expression Matching 2022 / SCA
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →