← ArchivePaper2018
Deep Appearance Models for Face Rendering
Deep generative model for face appearance and geometry enabling photorealistic rendering and synthesis of facial expressions.
Abstract
We introduce a deep appearance model for rendering the human face. Inspired by Active Appearance Models, we develop a data-driven rendering pipeline that learns a joint representation of facial geometry and appearance from a multiview capture setup. Vertex positions and view-specific textures are modeled using a deep variational autoencoder that captures complex nonlinear effects while producing a smooth and compact latent representation. View-specific texture enables the modeling of view-dependent effects such as specularity. In addition, it can also correct for imperfect geometry stemming from biased or low resolution estimates. This is a significant departure from the traditional graphics pipeline, which requires highly accurate geometry as well as all elements of the shading model to achieve realism through physically-inspired light transport. Acquiring such a high level of accuracy is difficult in practice, especially for complex and intricate parts of the face, such as eyelashes and the oral cavity. These are handled naturally by our approach, which does not rely on precise estimates of geometry. Instead, the shading model accommodates deficiencies in geometry though the flexibility afforded by the neural network employed. At inference time, we condition the decoding network on the viewpoint of the camera in order to generate the appropriate texture for rendering.
How to read this
- Category
- Method: deep generative model for face rendering
- Contributions
- Introduces a deep appearance model that jointly learns facial geometry and appearance from a multiview capture setup
- Models vertex positions and view-specific textures with a deep variational autoencoder, capturing view-dependent effects like specularity in a compact latent space
- Tolerates imperfect or low-resolution geometry by letting the shading model compensate, avoiding the traditional pipeline's need for highly accurate geometry
- Context
- Inspired by Active Appearance Models and related to data-driven face capture and reenactment (Thies et al., Face2Face), departing from physically-based light-transport rendering.Builds on: Face2Face: Real-Time Face Capture and Reenactment of RGB Videos
- Correctness
- Trained from a specific multiview capture rig; the key assumption is that view-specific texture can absorb geometric error, so results depend on capture coverage and the approach is demonstrated for face rendering rather than as a general light-transport replacement.
- Clarity
- Reasonably accessible given the AAM framing; a first pass conveys the idea, a second pass is needed for the VAE architecture and view conditioning.
- How to read it
- Focus on the joint geometry-plus-view-dependent-texture VAE and why precise geometry becomes unnecessary; do a second pass on the model details if rendering or capture is your focus.
Built upon by
Related work
- Facial Animation with Disentangled Identity and Motion using Transformers 2022 / SCA
- Learning a Generalized Physical Face Model From Data 2024 / SIGGRAPH
- CANRIG: Cross-Attention Neural Face Rigging with Variable Local Control 2026 / Eurographics
- Acquiring the Reflectance Field of a Human Face 2000 / SIGGRAPH
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →