Neural Volumes: Learning Dynamic Renderable Volumes from Images

How to read this

Category

Academic paper introducing a learned volumetric representation for dynamic capture

Contributions

• Represents a dynamic captured object as a learned 3D volume rather than a tracked mesh, sidestepping reconstruction and correspondence entirely
• Trains an encoder decoder end to end, supervised directly from calibrated multi view 2D images, with no explicit geometry or tracking supervision
• Renders through a differentiable ray marching operation inspired by the integral projection model used in tomographic imaging, which is what makes the image only supervision possible
• Handles the cases that break mesh based capture: thin structures, changing topology, translucency, scattering and occlusion, hair being the obvious example

Context

This descends directly from the same lab's Deep Appearance Models for Face Rendering, which still relied on a tracked mesh and a view dependent texture. Neural Volumes drops the mesh, and in doing so it sits at the hinge between classical performance capture and the neural rendering wave that NeRF and its descendants turned into a field. For a character person it is worth reading as the moment the representation itself stopped being geometry.Builds on: Deep Appearance Models for Face Rendering

Correctness

The claims rest on multi view captures of real subjects and objects, with the evidence being reconstruction and novel view quality rather than an animation benchmark. Note the honest limits for character work: this captures and replays an appearance, it does not give you a rig, and the volumetric representation is expensive and not directly editable by an animator. Its value to a rigger is conceptual and as an ancestor, not as a tool you would deploy.

Clarity

Clearly written for a graphics audience, and unusually readable for a neural rendering paper because the core idea, a differentiable version of tomographic projection, is explained in plain physical terms.

How to read it

First pass, read the abstract and Figure 1, then look at what specifically fails for mesh based capture, since that framing is the whole motivation. Second pass, work through the encoder decoder and the ray marching operator together, because the differentiability of the renderer is what allows training from images alone. A third pass is worth it only if you work on neural avatars, in which case follow the line forward into the Codec Avatar work rather than dwelling here.