← ArchivePaper2021

Real-time Deep Dynamic Characters

Marc Habermann, Lingjie Liu, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, Christian Theobalt

Real-time neural character deformation combining skeleton-driven skinning with learned dynamic detail including cloth and soft-tissue effects.

Open the paper ↗Trace in the lineage graph

Abstract

This paper proposes a deep videorealistic 3D human character model that displays highly realistic shape, motion, and dynamic appearance learned in a weakly supervised way from multi-view imagery. In contrast to prior work, the controllable character displays motion-dependent dynamics such as the swing of a skirt without requiring physics simulation, along with a learned dynamic texture model that captures motion-dependent appearance and view-dependent lighting effects. The method uses a parametric differentiable character representation combined with embedded deformation and per-vertex displacements regressed by a novel structure-aware graph convolutional network, and a neural generative dynamic texture model, all trained from multi-view video using differentiable rendering. Taking only a skeletal motion and camera view as input, the model creates physically plausible clothing deformations and video-realistic textures in real time.

How to read this

Category: Method: real-time learned dynamic character model (deformation plus appearance)
Contributions: Learns a controllable, videorealistic 3D human with motion-dependent dynamics (such as a swinging skirt) without physics simulation, weakly supervised from multi-view video
Combines a parametric differentiable character with embedded deformation and per-vertex displacements regressed by a structure-aware graph convolutional network
Adds a neural generative dynamic texture model for motion-dependent appearance and view-dependent lighting, producing real-time output from skeletal motion and camera view
Context: Advances learned deformation approximation in the spirit of Fast and Deep Deformation Approximations (Bailey et al.), extending it to clothing dynamics and a jointly learned dynamic texture trained via differentiable rendering.Builds on: Fast and Deep Deformation Approximations
Correctness: Trained weakly supervised from multi-view imagery and producing physically plausible (not simulated) deformations, so results depend on captured motions and views; plausibility rather than physical accuracy and generalization beyond the training distribution are the caveats to keep in mind.
Clarity: Dense, multi-component system; a first pass conveys the skeleton-plus-learned-dynamics-plus-dynamic-texture pipeline, second and third passes are needed for the graph-network and differentiable-rendering training details.
How to read it: First pass for the overall real-time architecture and what each module contributes; do a second pass on the structure-aware GCN and dynamic texture model if neural character rendering is your focus.

Builds on

Fast and Deep Deformation Approximations 2018

Built upon by

Nothing yet.

Related work

NeuroSkinning: Automatic Skin Binding for Production Characters with Deep Graph Networks 2019 / SIGGRAPH
Neural Volumes: Learning Dynamic Renderable Volumes from Images 2019 / SIGGRAPH
Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies 2021 / CVPR
Learning Skeletal Articulations with Neural Blend Shapes 2021 / SIGGRAPH

Keywords

deep dynamic characters graph convolutional network structure aware graph convolution dynamic texture differentiable rendering embedded deformation dual quaternion skinning weakly supervised learning multi-view video per-vertex displacement neural rendering spherical harmonics lighting cloth deformation dynacap silhouette loss as-rigid-as-possible

This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →