Wan-Dancer: A Hierarchical Framework for Minute-scale Coherent Music-to-Dance Generation

Source: arXiv:2607.09581 · Published 2026-07-10 · By Mingyang Huang, Peng Zhang, Li Hu, Guangyuan Wang, Bang Zhang

TL;DR

This paper addresses the challenge of generating long-duration (minute-scale), high-resolution, rhythmically synchronized dance videos from music input, an area where prior diffusion-based approaches are typically limited to under 20 seconds due to temporal coherence issues and computational complexity. The authors propose Wan-Dancer, a hierarchical global-to-local diffusion framework that decouples global choreographic planning from local frame-level refinement, using full-track musical context to ensure temporal stability and choreography consistency over extended sequences. Novel techniques include dynamic frame rate adaptation via absolute time-aware rotary positional embeddings (RoPE), an optical flow-based weighted loss to improve motion continuity, and motion speed stratification to preserve detail in rapid movements.

The model is trained on a proprietary 200-hour, high-resolution (720p/30fps) dataset spanning five dance genres and conditioned on multimodal inputs: music, textual prompts, and reference images. Extensive evaluations show Wan-Dancer surpasses two strong baselines—X-Dancer and MusicInfuser—across quantitative dance quality, video quality, and prompt alignment metrics, while also generating coherent videos up to 160 seconds long. Ablations confirm the importance of their hierarchical design, optical flow loss weighting, dynamic frame rate injection, and motion speed control. Additionally, a low-rank adaptation (LoRA) approach enables efficient customization to replicate specific choreographies with few-shot example videos. Overall, the paper presents a significant advance in coherent, long-form music-to-dance video synthesis.

Key findings

• Wan-Dancer generates stable 720p/30fps dance videos exceeding one minute (up to 160s) with preserved temporal coherence and identity, overcoming the ~20s barrier of prior works.
• In dance quality metrics (style alignment, beat synchronization, body representation, movement realism, choreography complexity), Wan-Dancer scores an average of 8.46, outperforming MusicInfuser (6.23) and X-Dancer (6.06).
• In video quality (imaging, aesthetic, temporal consistency), Wan-Dancer achieves an overall average score of 7.46, beating MusicInfuser (5.22) and X-Dancer (6.23).
• For prompt alignment (style capture, creative interpretation, satisfaction), Wan-Dancer attains 9.03 average vs MusicInfuser's 6.61, showing superior semantic controllability via text conditioning.
• Ablation studies demonstrate that global-to-local hierarchical modeling eliminates temporal drift and identity inconsistency seen in naive sequential clip stitching.
• Incorporating an optical flow loss weight reduces motion blur during rapid movements, improving edge fidelity and structural accuracy in fast dance segments.
• Dynamic frame rate injection into RoPE embeddings enables flexible temporal alignment with varying music durations, preserving smooth choreography at frame rates between 3 and 15 fps.
• Motion speed stratification focusing on medium-speed motions yields better choreographic realism and reduces artifacts compared to slow or fast-speed sampling.

Threat model

n/a — This work focuses on generative AI models for music-to-dance video synthesis rather than security or adversarial threat scenarios.

Methodology — deep read

1. Threat Model & Assumptions: The adversary model is not explicitly defined as the focus is generative video synthesis, but the framework assumes access to paired music, text prompts, and reference images to generate realistic dance videos. The adversary cannot arbitrarily manipulate temporal dependencies without degradation due to model design.

2. Data: The authors curated a proprietary dataset of approximately 200 hours of high-quality dance videos at minimum 720p/30fps, spanning five dance genres: Chinese Classical, K-Pop, Latin, Tap, and Street dance. Data segmentation divides raw videos into overlapping 5-second clips with 50% overlap for training. Audio features are extracted using Librosa. Optical flow masks are computed by SEA-RAFT for loss weighting. The dataset is stratified into slow (10%), medium (80%), and high (10%) velocity motion tiers based on kinematic velocity derived from keypoints.

3. Architecture / Algorithm: The model is based on a unified diffusion transformer network (DiT) with VAE encoding/decoding of video latents. Input includes concatenated video latents and a dynamic keyframe mask to indicate temporal conditioning: global phase masks only the first frame keyframe, forcing long-term global choreographic planning, while local phase randomly masks keyframes to learn frame interpolation and short-term temporal refinement. Rotary Positional Embeddings (RoPE) are enhanced with absolute time mappings for dynamic frame rate adaptation. The music encoder is a lightweight model extracting latent acoustic features injected into DiT blocks via dedicated music blocks for audio-visual alignment. Conditioning also includes CLIP visual features from a reference image and textual prompts encoded by umT5. The loss function predicts velocity fields between noisy and clean latent frames, weighted by optical flow-based spatial attention to emphasize motion continuity.

4. Training Regime: Training proceeds in two stages. First, low-resolution pretraining (320×544p) runs for 20,000 steps on 128 A100 GPUs with a learning rate of 1e-5 to establish rhythmic alignment and choreographic structure, applying optical flow loss only in local training. Second, high-resolution fine-tuning at native 720p resolution uses Unified Sequence Parallelism over 128 A100 GPUs for 4000 steps with the same learning rate to refine visual detail and temporal fidelity. For choreography customization, Low-Rank Adaptation (LoRA) fine-tuning is applied with rank 32 for 800 steps using 16 reference videos, enabling fast, targeted style mimicry.

5. Evaluation Protocol: They evaluate on Dance Quality (style alignment, beat alignment, body representation, movement realism, choreography complexity), Video Quality (imaging, aesthetic appeal, temporal consistency), and Prompt Alignment (style capture, creative interpretation, user satisfaction). Comparisons are conducted versus state-of-the-art end-to-end baselines X-Dancer and MusicInfuser, primarily using evaluation protocols from MusicInfuser for consistency. Ablation studies isolate contributions of global-to-local hierarchy, optical flow loss, dynamic frame rate injection, and motion speed stratification. Diversity is tested by varying music, reference images, and random seeds.

6. Reproducibility: The data is proprietary and not publicly released, limiting direct reproducibility. Implementation details on training schedules, hardware (128 A100 GPUs), hyperparameters (learning rates), and architecture components are disclosed. Source code or pretrained weights release is not mentioned, so full open reproducibility is unclear.

Concrete End-to-End Example: At inference, the input music, text prompt, reference image, random noise, dynamic FPS and keyframe mask feed into the global DiT network producing ~38 sparse global keyframes capturing coarse choreography over the full sequence. The sequence is segmented into overlapping local clips centered on each keyframe, with corresponding sliced audio segments feeding into the local DiT stage, which interpolates and refines frames producing high-fidelity, temporally consistent output videos at 720p/30fps. Finally, local clips concatenate into a coherent minute-scale video retaining rhythm alignment, visual detail, and identity consistency.

Technical innovations

• A hierarchical global-to-local diffusion framework decoupling long-range choreographic planning from local temporal refinement to overcome temporal drift in minute-scale dance video generation.
• Dynamic frame rate adaptation via time-mapped Rotary Positional Embeddings (RoPE) enabling precise temporal alignment with variable music durations and controllable frame density.
• An optical flow-based loss weighting scheme incorporating SEA-RAFT motion cues to enhance motion continuity and prevent blur during rapid movements.
• Motion speed stratification in training data categorization allowing the model to focus on medium-speed dynamics, improving choreography realism and reducing artifacts.
• Lightweight Low-Rank Adaptation (LoRA) for efficient customization of dance styles with few reference examples, avoiding costly full retraining.

Datasets

• Proprietary Alibaba music-to-dance dataset — ~200 hours, 720p/30fps, five genres (Chinese Classical, K-Pop, Latin, Tap, Street), source: internal and curated high-quality videos

Baselines vs proposed

• MusicInfuser [9] dance quality average = 6.23 vs Wan-Dancer = 8.46
• X-Dancer [3] dance quality average = 6.06 vs Wan-Dancer = 8.46
• MusicInfuser [9] video quality overall avg = 5.22 vs Wan-Dancer = 7.46
• X-Dancer [3] video quality overall avg = 6.23 vs Wan-Dancer = 7.46
• MusicInfuser [9] prompt alignment overall avg = 6.61 vs Wan-Dancer = 9.03

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.09581.

Fig 3: Minute-scale Video Generation. Our framework enables the synthesis of dance

Fig 4: Customized Choreography. Leveraging Low-Rank Adaptation (LoRA), our

Fig 2: Inference Pipeline. Initially, the global stage synthesizes a sparse keyframe

Fig 5: Comparison with X-Dancer [3] and MusicInfuser [9].

Fig 5 (page 9).

Fig 6 (page 9).

Fig 7 (page 9).

Fig 8 (page 9).

Limitations

• No explicit evaluation or mechanism for maintaining facial identity consistency across very long sequences; planned future work suggesting integrating face-embedding constraints.
• Proprietary dataset is not publicly available, limiting reproducibility and external benchmarking.
• Semantic alignment between music, text, and generated dance still has room for improvement; authors suggest enhanced multimodal semantic encoders as future work.
• Current architecture models only solo dancing scenarios; multi-dancer coordinated choreography remains unaddressed.
• The large compute requirements (128 A100 GPUs) may restrict practical accessibility for some researchers or practitioners.
• Temporal sampling resolution tradeoffs caused by dynamic frame rate injection are explored, but optimal strategies for varying genres or music complexity are not deeply analyzed.

Open questions / follow-ons

• How can facial and identity consistency be robustly maintained over minute-scale sequences, especially for close-up and multi-view scenarios?
• What improvements in multimodal semantic understanding can enhance choreography expressivity and user-controllable nuances in long-form synthesis?
• How can this hierarchical framework be extended effectively to multi-dancer or group choreography preserving inter-person spatial and temporal coordination?
• What tradeoffs exist in dynamic frame rate adaptation when handling highly complex or irregular musical rhythms, and can adaptive scheduling strategies improve results?

Why it matters for bot defense

While this paper is not directly about bot defense or CAPTCHA, its hierarchical framework and techniques for maintaining long-range temporal coherence in generative video diffusion models have conceptual relevance. Bot-defense systems leveraging generative video or animation to detect anomalies could benefit from understanding how to maintain temporal consistency over long sequences without degradation. Their approach to multimodal conditioning (audio, text, and visual prompts) and controlling fine-grained motion details might inspire methods to generate or analyze realistic human-like motions in security challenges or behavioral CAPTCHAs. Additionally, the introduction of dynamic frame rate injection and optical flow-guided loss could inform defenses that analyze motion smoothness and coherence patterns to differentiate bots from humans. Researchers working on synthetic media detection or dynamic CAPTCHA challenges involving video-based human activity simulation may find architectural and loss-function insights applicable to improving realism or detecting artifacts.

Cite

@article{arxiv2607_09581,
  title={ Wan-Dancer: A Hierarchical Framework for Minute-scale Coherent Music-to-Dance Generation },
  author={ Mingyang Huang and Peng Zhang and Li Hu and Guangyuan Wang and Bang Zhang },
  journal={arXiv preprint arXiv:2607.09581},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.09581}
}

Read the full paper

• arXiv:2607.09581 (abstract)
• arXiv:2607.09581 (PDF)