← ArchivePaper2024
SPARK: Self-supervised Personalized Real-time Monocular Face Capture
Kelian Baert, Shrisha Bharadwaj, Fabien Castan, Benoit Maujean, Marc Christie, Victoria Abrevaya, Adnane Boukhayma
Self-supervised personalized face capture reconstructing a relightable avatar from multiple unconstrained videos, with real-time tracking of unseen footage.
Abstract
Feedforward monocular face capture methods seek to reconstruct posed faces from a single image of a person. Current state of the art approaches have the ability to regress parametric 3D face models in real-time across a wide range of identities, lighting conditions and poses by leveraging large image datasets of human faces. These methods however suffer from clear limitations in that the underlying parametric face model only provides a coarse estimation of the face shape, thereby limiting their practical applicability in tasks that require precise 3D reconstruction (aging, face swapping, digital make-up,...). In this paper, we propose a method for high-precision 3D face capture taking advantage of a collection of unconstrained videos of a subject as prior information. Our proposal builds on a two stage approach. We start with the reconstruction of a detailed 3D face avatar of the person, capturing both precise geometry and appearance from a collection of videos. We then use the encoder from a pre-trained monocular face reconstruction method, substituting its decoder with our personalized model, and proceed with transfer learning on the video collection. Using our pre-estimated image formation model, we obtain a more precise self-supervision objective, enabling improved expression and pose alignment.
How to read this
- Category
- Method: personalized monocular face capture and avatar reconstruction
- Contributions
- SPARK, a self-supervised method for high-precision personalized 3D face capture from a collection of unconstrained videos
- A two-stage approach: reconstruct a detailed relightable avatar (geometry and appearance), then transfer-learn a personalized decoder onto a pretrained monocular encoder
- Real-time tracking of unseen footage with the personalized model
- Context
- A personalization layer over feedforward monocular face reconstruction, building on Learning an Animatable Detailed 3D Face Model from In-The-Wild Images (DECA, Feng 2021).Builds on: Learning an Animatable Detailed 3D Face Model from In-The-Wild Images
- Correctness
- Aims to overcome the coarse shape of generic parametric models by exploiting per-subject video priors; precision therefore depends on having a suitable collection of videos per subject, a requirement to keep in mind versus single-image methods.
- Clarity
- Fairly technical; a first pass conveys the two-stage personalize-then-track idea, a second pass for the encoder swap and transfer-learning details.
- How to read it
- First pass for the avatar-then-tracker structure and why personalization beats a generic parametric model; second pass on the encoder substitution and transfer learning if you plan to reproduce it.
Built upon by
Nothing yet.
Related work
- EMOCA: Emotion Driven Monocular Face Capture and Animation 2022 / CVPR
- I M Avatar: Implicit Morphable Head Avatars from Videos 2022 / CVPR
- FLARE: Fast Learning of Animatable and Relightable Mesh Avatars 2023 / SIGGRAPH Asia
- Learning a Model of Facial Shape and Expression from 4D Scans 2017 / SIGGRAPH Asia
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →