← ArchivePaper2019
Neural State Machine for Character-Scene Interactions
Neural state machine for character-scene interaction synthesis generating contextual motions like sitting, picking up objects, and navigating environments.
Abstract
We propose Neural State Machine, a novel data-driven framework to guide characters to achieve goal-driven actions with precise scene interactions. Even a seemingly simple task such as sitting on a chair is notoriously hard to model with supervised learning. This difficulty is because such a task involves complex planning with periodic and non-periodic motions reacting to the scene geometry to precisely position and orient the character. Our proposed deep auto-regressive framework enables modeling of multi-modal scene interaction behaviors purely from data. Given high-level instructions such as the goal location and the action to be launched there, our system computes a series of movements and transitions to reach the goal in the desired state. To allow characters to adapt to a wide range of geometry such as different shapes of furniture and obstacles, we incorporate an efficient data augmentation scheme to randomly switch the 3D geometry while maintaining the context of the original motion. To increase the precision to reach the goal during runtime, we introduce a control scheme that combines egocentric inference and goal-centric inference.
How to read this
- Category
- Method: a data-driven neural framework for character-scene interaction synthesis
- Contributions
- Proposes the Neural State Machine, a deep auto-regressive framework that synthesizes goal-driven motions with precise scene interactions (sitting, picking up objects, navigation)
- Introduces a data augmentation scheme that randomly switches 3D geometry while preserving motion context, so characters adapt to varied furniture and obstacles
- Adds a runtime control scheme combining egocentric and goal-centric inference to improve goal-reaching precision
- Context
- Builds on data-driven neural motion controllers, specifically the Mode-Adaptive Neural Networks for quadruped motion control (Zhang et al. 2018), extending that lineage to scene-aware human interactions.Builds on: Mode-Adaptive Neural Networks for Quadruped Motion Control
- Correctness
- Demonstrated on multi-modal interaction behaviors learned purely from data; precise contact and goal-state placement is the hard part the authors target, and quality depends on captured interaction data and the augmentation covering the target scene geometry.
- Clarity
- Dense but well-motivated; a first pass conveys the goal-driven interaction idea, a second pass is needed for the auto-regressive architecture and the dual-inference control.
- How to read it
- First pass for the interaction-synthesis idea and the geometry-augmentation trick; second pass on the network structure and egocentric/goal-centric blending if you intend to reimplement or build scene-aware controllers.
Built upon by
- Local Motion Phases for Learning Multi-Contact Character Movements 2020
- DeepPhase: Periodic Autoencoders for Learning Motion Phase Manifolds 2022
- Machine Learning Summit: 4 Years of Bringing Characters to Life with Computer Brains 2022
- The Wanderings of Odysseus in 3D Scenes 2022
- IMoS: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactions 2023
- Synthesizing Physical Character-Scene Interactions 2023
Related work
- Local Motion Phases for Learning Multi-Contact Character Movements 2020 / SIGGRAPH
- Phase-Functioned Neural Networks for Character Control 2017 / SIGGRAPH
- Learned Motion Matching 2020 / SIGGRAPH
- Mode-Adaptive Neural Networks for Quadruped Motion Control 2018 / SIGGRAPH
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →