← ArchivePaper2023

EDGE: Editable Dance Generation from Music

Jonathan Tseng, Rodrigo Castellon, C. Karen Liu

Transformer diffusion model paired with Jukebox audio features for physically plausible, editable music-driven dance generation.

Open the paper ↗Trace in the lineage graph

Abstract

Dance is an important human art form, but creating new dances can be difficult and time-consuming. In this work, we introduce Editable Dance GEneration (EDGE), a state-of-the-art method for editable dance generation that is capable of creating realistic, physically-plausible dances while remaining faithful to the input music. EDGE uses a transformer-based diffusion model paired with Jukebox, a strong music feature extractor, and confers powerful editing capabilities well-suited to dance, including joint-wise conditioning, and in-betweening. We introduce a new metric for physical plausibility, and evaluate dance quality generated by our method extensively through (1) multiple quantitative metrics on physical plausibility, beat alignment, and diversity benchmarks, and more importantly, (2) a large-scale user study, demonstrating a significant improvement over previous state-of-the-art methods. Qualitative samples from our model can be found at our website.

How to read this

Category: Method: music-driven dance generation (transformer diffusion)
Contributions: EDGE, a transformer-based diffusion model conditioned on Jukebox music features for realistic, physically plausible dance
Editing capabilities suited to dance, including joint-wise conditioning and in-betweening
A new physical-plausibility metric, plus quantitative benchmarks and a large-scale user study
Context: Sits in the motion-diffusion lineage, building on the Human Motion Diffusion Model (tevet-mdm-2022) and pairing it with a strong pretrained music feature extractor (Jukebox) for the music-to-motion task.Builds on: Human Motion Diffusion Model
Correctness: Plausibility is judged by a newly introduced metric plus a user study, so weigh the proposed metric against the subjective study rather than treating it as ground truth; physical plausibility here is a learned, evaluated proxy, not a simulated guarantee.
Clarity: Accessible at a high level; a first pass conveys the conditioning-plus-editing idea, do a second pass for the diffusion formulation and metric definition.
How to read it: Focus on the editing primitives (joint-wise conditioning, in-betweening) and the physical-plausibility metric; a second pass pays off if you care about how the metric is defined and validated against the user study.

Builds on

Human Motion Diffusion Model 2022

Built upon by

Nothing yet.

Related work

Human Motion Diffusion Model 2022 / arXiv
Executing Your Commands via Motion Diffusion in Latent Space 2023 / CVPR
MotionCLIP: Exposing Human Motion Generation to CLIP Space 2022 / CVPR
CLoSD: Closing the Loop between Simulation and Diffusion for Multi-Task Character Control 2024 / arXiv

Keywords

editable dance generation music-conditioned generation diffusion model transformer decoder jukebox contact consistency loss physical foot contact score foot sliding classifier-free guidance motion in-betweening joint-wise conditioning smpl 6dof rotation aist++long-form generation cross-attention beat alignment

This page summarises the entry and links to its original source. The archive never hosts or redistributes the publication itself.Show it in the full archive list →