← ArchivePaper2024
Dog Code: Human to Quadruped Embodiment Using Shared Codebooks
Dónal Egan, Alberto Jovane, Jan Szkaradek, George Fletcher, Darren Cosker, Rachel McDonnell
Shared finite-scalar-quantisation codebook maps retargeted human skeleton motion to realistic synchronised quadruped animation.
Abstract
Many VR animal embodiment sytsems suffer from poor animation fidelity, typically animating the animal avatars using inverse kinematics. We address this issue, presenting a novel deep-learning method, centred around a shared codebook, for mapping human motion to quadruped motion. Rather than trying to directly bridge the gap from human motion to quadruped motion, a task which has proven difficult, we first use a rule-based retargeter, relying on inverse and forward kinematics, to retarget human motions to an intermediate motion domain in which the motions share the same skeleton as the quadruped. We then use finite scalar quantization to construct a shared latent space, or codebook, between this intermediate domain and the quadruped motion domain. We do this by first pre-defining a finite number of discrete latent codes and then teaching these codes, using unsupervised deep-learning, to represent semantically similar motions in the two domains. We incorporate our real-time human-to-quadruped motion mapping into a VR quadruped embodiment system. The output quadruped animations are natural and realistic, while also preserving the semantics of users’ actions. Moreover, there is a strong synchrony between the input human motions and retargeted quadruped motions, an important factor for inducing a strong sense of VR embodiment.
How to read this
- Category
- Method: deep-learning human-to-quadruped motion mapping for VR embodiment
- Contributions
- A shared codebook (via finite scalar quantization) linking an intermediate human-skeleton domain to the quadruped motion domain
- A two-stage approach: rule-based IK/FK retargeting of human motion to an intermediate domain sharing the quadruped skeleton, then learned mapping to quadruped motion
- Real-time integration into a VR quadruped embodiment system producing natural, semantics-preserving animations
- Context
- Advances neural quadruped animation and motion retargeting, building on the authors' prior 'How to Train Your Dog: Neural Enhancement of Quadruped Animations'.Builds on: How to Train Your Dog: Neural Enhancement of Quadruped Animations
- Correctness
- Outputs are reported as natural and realistic while preserving action semantics; keep in mind the pipeline relies on a rule-based intermediate retargeter and an unsupervised shared codebook, so fidelity depends on that intermediate domain and on the captured quadruped data, and the abstract's quantitative claims are truncated.
- Clarity
- Accessible if familiar with motion retargeting and VQ-style latent spaces; a first pass conveys the two-stage idea, a second pass for the finite scalar quantization and codebook training.
- How to read it
- Focus on why an intermediate shared-skeleton domain plus a learned shared codebook beats direct human-to-quadruped mapping; a second pass pays off for the quantization and unsupervised training details.
Built upon by
Nothing yet.
Related work
Keywords
This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →