← ArchivePaper2023
FLARE: Fast Learning of Animatable and Relightable Mesh Avatars
Shrisha Bharadwaj, Yufeng Zheng, Otmar Hilliges, Michael J. Black, Victoria Fernandez-Abrevaya
Creates animatable, relightable mesh avatars from monocular video using differentiable rasterization and learned blendshapes, compatible with standard rendering pipelines.
Abstract
Our goal is to efficiently learn personalized animatable 3D head avatars from videos that are geometrically accurate, realistic, relightable, and compatible with current rendering systems. While 3D meshes enable efficient processing and are highly portable, they lack realism in terms of shape and appearance. Neural representations, on the other hand, are realistic but lack compatibility and are slow to train and render. Our key insight is that it is possible to efficiently learn high-fidelity 3D mesh representations via differentiable rendering by exploiting highly-optimized methods from traditional computer graphics and approximating some of the components with neural networks. To that end, we introduce FLARE, a technique that enables the creation of animatable and relightable mesh avatars from a single monocular video. First, we learn a canonical geometry using a mesh representation, enabling efficient differentiable rasterization and straightforward animation via learned blendshapes and linear blend skinning weights. Second, we follow physically-based rendering and factor observed colors into intrinsic albedo, roughness, and a neural representation of the illumination, allowing the learned avatars to be relit in novel scenes. Since our input videos are captured on a single device with a narrow field of view, modeling the surrounding environment light is non-trivial.
How to read this
- Category
- Method: animatable, relightable mesh head avatars from monocular video
- Contributions
- FLARE, building animatable and relightable mesh head avatars from a single monocular video
- A canonical mesh geometry learned via differentiable rasterization, animated with learned blendshapes and linear blend skinning
- Physically-based appearance factoring colors into albedo, roughness, and a neural illumination representation, keeping pipeline compatibility
- Context
- Responds to monocular neural-avatar work such as Neural Head Avatars from Monocular RGB Videos (neural-head-avatars-grassal-2022), but favors a mesh representation for efficiency, portability, and renderer compatibility.Builds on: Neural Head Avatars from Monocular RGB Videos
- Correctness
- The key insight is approximating some components with neural networks while keeping an efficient mesh and differentiable rasterization; relightability rests on the PBR factoring, so the albedo/roughness/illumination split is an approximation whose quality is demonstrated on monocular captures rather than guaranteed under arbitrary lighting.
- Clarity
- Reasonably accessible if you know differentiable rendering and PBR; a first pass conveys the mesh-versus-neural trade-off, a second pass for the factoring and training.
- How to read it
- Focus on why a mesh is chosen over a neural field (speed, compatibility) and on the intrinsic decomposition; a second pass pays off for the differentiable rasterization and relighting details.
Builds on
Built upon by
Nothing yet.
Related work
- PointAvatar: Deformable Point-Based Head Avatars from Videos 2023 / CVPR
- Learning an Animatable Detailed 3D Face Model from In-The-Wild Images 2021 / SIGGRAPH
- Neural Head Avatars from Monocular RGB Videos 2022 / CVPR
- SPARK: Self-supervised Personalized Real-time Monocular Face Capture 2024 / SIGGRAPH Asia
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →