Improving Facial Rig Semantics for Tracking and Retargeting

How to read this

Category

research paper, facial performance tracking and retargeting via a shared, semantically calibrated rig

Contributions

• Proposes retargeting a tracked facial performance to another person or a game or VR character by using the same rig framework, 3DMM, FLAME, MetaHuman, and so on, for both performer and target, removing the need to map semantics between different rig frameworks
• Uses volumetric morphing to fit that shared rig framework to both performer and target, then calibrates each fitted rig's motion signature with a carefully chosen set of Simon Says expressions
• Shows that even well calibrated rigs of this kind still produce undesirable, non semantic animation controls when tracking a live performance, despite producing acceptable reconstructed geometry, since a well behaved function can have an arbitrarily ill conditioned inverse
• Proposes a fine tuning approach based on implicit differentiation that treats the tracker as a potentially non differentiable black box, adjusting expression parameters so tracker output matches semantically meaningful animation controls
• Reformulates Broyden's method to connect it to more robust finite difference estimates, improving the numerical search direction used in the implicit differentiation scheme

Context

This paper sits in the facial performance retargeting literature it surveys extensively, citing Zhu and Joslin 2024 for a review, and it builds most directly on volumetric morphing and Simon Says calibration techniques from Zhu et al 2024 and on prior work personalizing FLAME expression coefficients from Baert et al 2024. No builds_on ancestors are listed in the archive dataset. Its distinguishing move against most retargeting work, which focuses on mapping geometry or rig controls between different frameworks, is to sidestep that mapping problem entirely by insisting performer and target share one rig framework, then fixing the residual problem that a well calibrated rig can still yield a poorly conditioned tracker.

Correctness

The paper backs its claims with a mathematical linearization analysis showing a tracker is a geometry reconstruction followed by the inverse of a rig operation, and it reports the fine tuning approach validated on real world problems with domain expert feedback from animators and riggers. It is explicit that its own well calibrated rigs can still produce undesirable controls before fine tuning, which is the core limitation the paper is patching rather than eliminating.

Clarity

A math heavy paper, implicit function theorem, a reformulation of Broyden's method, aimed at technical riggers and computer vision researchers comfortable with inverse problem formalism, though the practical motivation for why calibrated rigs still track badly is explained clearly in prose before the equations arrive.

How to read it

Five minute pass: read the abstract and the four contribution bullets at the end of Section 1 to get the shape of the argument, shared rig framework, Simon Says calibration, then implicit differentiation fine tuning of the tracker. Second pass: read Sections 1 and 3 closely for the V, S, R, T notation and the combined loss in Equation 3, since that formalism is reused throughout. Third pass is worth it for riggers or tracking engineers directly building retargeting pipelines on parametric rigs like FLAME or MetaHuman, since the later sections on the Simon Says expression set and the Broyden's method reformulation are the concrete, implementable pieces.