← ArchivePaper2023
IMoS: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactions
Anindita Ghosh, Rishabh Dabral, Vladislav Golyanik, Christian Theobalt, Philipp Slusallek
First framework to synthesize full-body character motion with 3D objects from text-based intent labels, supporting single and two-handed interactions.
Abstract
Can we make virtual characters in a scene interact with their surrounding objects through simple instructions? Is it possible to synthesize such motion plausibly with a diverse set of objects and instructions? Inspired by these questions, we present the first framework to synthesize the full‐body motion of virtual human characters performing specified actions with 3D objects placed within their reach. Our system takes textual instructions specifying the objects and the associated ‘intentions’ of the virtual characters as input and outputs diverse sequences of full‐body motions. This contrasts existing works, where full‐body action synthesis methods generally do not consider object interactions, and human‐object interaction methods focus mainly on synthesizing hand or finger movements for grasping objects. We accomplish our objective by designing an intent‐driven full‐body motion generator, which uses a pair of decoupled conditional variational auto‐regressors to learn the motion of the body parts in an autoregressive manner. We also optimize the 6‐DoF pose of the objects such that they plausibly fit within the hands of the synthesized characters. We compare our proposed method with the existing methods of motion synthesis and establish a new and stronger state‐of‐the‐art for the task of intent‐driven motion synthesis.
How to read this
- Category
- Method: intent-driven full-body human-object interaction synthesis
- Contributions
- First framework to synthesize full-body character motion interacting with 3D objects from text-based intent labels
- An intent-driven generator using a pair of decoupled conditional variational auto-regressors to learn body-part motion autoregressively
- A 6-DoF object pose optimization so objects plausibly fit within the synthesized character's hands, supporting single and two-handed interactions
- Context
- Relates to character-scene interaction work (Neural State Machine) and bridges full-body action synthesis with hand/finger grasping methods, which it argues had been treated separately.Builds on: Neural State Machine for Character-Scene Interactions
- Correctness
- Plausibility hinges on the decoupled VAE auto-regressors and the post-hoc object-fit optimization; diversity and physical correctness are bounded by the training data and the set of objects/intents used, so contact and penetration quality outside that set is uncertain.
- Clarity
- Readable framing with a clear motivating question; a first pass conveys the pipeline, a second pass is needed for the auto-regressor coupling and object-pose optimization.
- How to read it
- First pass for the problem setup and the decoupled-generator idea; second pass on how object 6-DoF fitting is coupled to the body motion if you plan to reuse the interaction model.
Builds on
Built upon by
Nothing yet.
Related work
- MotionCLIP: Exposing Human Motion Generation to CLIP Space 2022 / CVPR
- T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations 2023 / CVPR
- Learning Robust and Scalable Motion Matching with Lipschitz Continuity and Sparse Mixture of Experts 2023 / MIG
- TEMOS: Generating Diverse Human Motions from Textual Descriptions 2022 / CVPR
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →