← ArchivePaper2015
Driving High-Resolution Facial Scans with Video Performance Capture
Graham Fyffe, Andrew Jones, Oleg Alexander, Ryosuke Ichikari, Paul Debevec
Drives sub-millimeter facial scans from video-based performance capture by solving blendshape coefficients from tracked video features.
Abstract
We present a process for rendering a realistic facial performance with control of viewpoint and illumination. The performance is based on one or more high-quality geometry and reflectance scans of an actor in static poses, driven by one or more video streams of a performance. We compute optical flow correspondences between neighboring video frames, and a sparse set of correspondences between static scans and video frames. The latter are made possible by leveraging the relightability of the static 3D scans to match the viewpoint(s) and appearance of the actor in videos taken in arbitrary environments. As optical flow tends to compute proper correspondence for some areas but not others, we also compute a smoothed, per-pixel confidence map for every computed flow, based on normalized cross-correlation. These flows and their confidences yield a set of weighted triangulation constraints among the static poses and the frames of a performance. Given a single artist-prepared face mesh for one static pose, we optimally combine the weighted triangulation constraints, along with a shape regularization term, into a consistent 3D geometry solution over the entire performance that is drift free by construction. In contrast to previous work, even partial correspondences contribute to drift minimization, for example, where a successful match is found in the eye region but not the mouth.
How to read this
- Category
- Method: video-driven facial performance reconstruction
- Contributions
- A pipeline that drives high-quality static geometry and reflectance scans of an actor from one or more performance video streams, with control over viewpoint and illumination
- Optical-flow correspondences between video frames plus sparse scan-to-video correspondences, the latter enabled by relighting the static scans to match arbitrary video appearance
- A per-pixel flow confidence map (from normalized cross-correlation) feeding weighted triangulation constraints, solved with shape regularization into a drift-free 3D geometry over the whole performance
- Context
- Builds on light-stage reflectance-field acquisition (Debevec et al., Acquiring the Reflectance Field of a Human Face) to make static scans relightable and matchable to in-the-wild performance video.Builds on: Acquiring the Reflectance Field of a Human Face
- Correctness
- The drift-free claim rests on combining many weighted, confidence-gated correspondences with regularization; reliability therefore depends on flow quality and confidence estimation, which can be weak in low-texture or occluded face regions.
- Clarity
- Moderately technical; a first pass conveys the relight-and-correspond idea, but the constraint formulation needs a second pass.
- How to read it
- Read first for the relightable-scan-to-video matching trick and the confidence map; second pass on the weighted triangulation/regularization solve if you need the math.
Builds on
Built upon by
Nothing yet.
Related work
- FaceLab: Scalable Facial Performance Capture for Visual Effects 2020 / DigiPro
- Real-Time High-Fidelity Facial Performance Capture 2015 / SIGGRAPH
- High Resolution Passive Facial Performance Capture 2010 / SIGGRAPH
- Facial Retargeting with Automatic Range of Motion Alignment 2017 / SIGGRAPH
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →