← ArchivePaper2023
NOFA: NeRF-based One-shot Facial Avatar Reconstruction
Yuda Hong, Zhihao Li, Yaxin Chen, Xuaner Zhang, Mingtao Feng, Yuefang Gao
Reconstructs a high-fidelity 3D facial avatar from a single source image using NeRF, enabling photo-realistic reenactment driven by arbitrary target faces.
Abstract
3D facial avatar reconstruction has been a significant research topic in computer graphics and computer vision, where photo-realistic rendering and flexible controls over poses and expressions are necessary for many related applications. Recently, its performance has been greatly improved with the development of neural radiance fields (NeRF). However, most existing NeRF-based facial avatars focus on subject-specific reconstruction and reenactment, requiring multi-shot images containing different views of the specific subject for training, and the learned model cannot generalize to new identities, limiting its further applications. In this work, we propose a one-shot 3D facial avatar reconstruction framework that only requires a single source image to reconstruct a high-fidelity 3D facial avatar. For the challenges of lacking generalization ability and missing multi-view information, we leverage the generative prior of 3D GAN and develop an efficient encoder-decoder network to reconstruct the canonical neural volume of the source image, and further propose a compensation network to complement facial details. To enable fine-grained control over facial dynamics, we propose a deformation field to warp the canonical volume into driven expressions. Through extensive experimental comparisons, we achieve superior synthesis results compared to several state-of-the-art methods.
How to read this
- Category
- Method: one-shot NeRF-based facial avatar reconstruction
- Contributions
- A framework that reconstructs a high-fidelity 3D facial avatar from a single source image
- Use of a 3D GAN generative prior with an encoder-decoder to recover a canonical neural volume, plus a compensation network for facial detail
- A deformation component enabling fine-grained pose/expression control and reenactment driven by arbitrary target faces
- Context
- Relates to animatable detailed face models from in-the-wild images such as DECA (Feng 2021), addressing the subject-specific, multi-shot limitation of prior NeRF avatars.Builds on: Learning an Animatable Detailed 3D Face Model from In-The-Wild Images
- Correctness
- Tackles the lack of multi-view info from a single image by leaning on a 3D-GAN prior; one-shot quality is therefore bounded by that prior's coverage, so identities far from its distribution are a caution.
- Clarity
- Accessible in motivation; a first pass conveys the one-shot pipeline, a second pass clarifies the canonical-volume and compensation-network mechanics.
- How to read it
- First pass for how the GAN prior plus compensation network enable single-image avatars; second pass on the deformation/control module if you need driven reenactment.
Built upon by
Nothing yet.
Related work
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →