Dense-Sparse Dynamic Time Warping for Customizing Piano Concerto Accompaniments

Source: arXiv:2607.18189 · Published 2026-07-20 · By TJ Tsai, Kavi Dey, Yigitcan Ozer, Meinard Muller

TL;DR

This paper addresses the problem of customizing prerecorded orchestral accompaniments in Music Minus One (MMO) recordings to better synchronize with a pianist's personal playing style. Traditional automated accompaniment systems often rely on symbolic music scores, which are scarce or unavailable for many piano concertos. Instead, the authors propose a score-free, audio-based alignment approach that leverages three types of audio data: user-recorded solo piano (Puser), orchestral-only MMO recordings (Oacc), and mixed piano-orchestra recordings from public sources like YouTube (Mref). The mixed recording serves as a crucial intermediary to align the solo and orchestral parts, enabling time-scale modifications of Oacc to match the user's tempo. The key technical challenge arises from spectral mismatch across these audio domains, due to differing instrumentations and recording conditions.

To overcome this, the authors introduce Dense-Sparse Dynamic Time Warping (DS-DTW), a novel alignment algorithm that selectively focuses on a sparse subset of audio frames exhibiting prominent timing cues, reducing sensitivity to regions with spectral mismatch such as silence or rests. They collected and annotated a benchmark dataset spanning four piano concerto movements, involving MMO orchestra-only tracks, piano-only recordings by recruited pianists, and mix recordings from IMSLP. Experimental results show that DS-DTW outperforms naive and iterative subtractive alignment baselines and matches or surpasses complex source separation-based approaches (including Spleeter and HDemucs) in alignment accuracy, all while being conceptually simpler and having no trainable parameters. DS-DTW reduces alignment error rates by up to 78% at 2 second tolerance compared to standard pairwise DTW. This work provides a practical and robust framework for offline piano concerto accompaniment customization without requiring symbolic scores or complex source separation.

Key findings

• DS-DTW reduces Pquery–Oacc alignment error rate from 2.7% to 0.6% at 2 second error tolerance compared to naive pairwise DTW on the Concerto Accompaniment Benchmark (Table II).
• At 1 second error tolerance, DS-DTW achieves 2.9% error rate versus 5.1% for naive DTW, a 43% error reduction.
• Source separation + DTW using HDemucs yields 0.3% error at 2s tolerance, slightly better than DS-DTW's 0.6%, but requires specialized trained models.
• Iterative Subtractive Alignment (ISA) performs worse than naive DTW, with 7.5% error at 2s tolerance.
• DS-DTW tuning of hyperparameter γ (fraction of features kept) between 0.6 and 0.9 outperforms standard DTW (γ=1.0), with best results at γ=0.8 (Table III).
• The benchmark dataset contains 4 concertos, 4 orchestra-only MMO recordings, 4 piano-only user recordings, 8 mix recordings totaling nearly 3 hours (Table I).
• Using Mref as an intermediary for alignment helps overcome lack of symbolic score and severe spectral mismatch.
• Selecting audio frames based on spectral flux to focus alignment only on high-flux (timing cue) frames improves robustness.

Threat model

The scenario assumes an offline environment where the adversary is the natural spectral mismatch and timing variations across three types of audio recordings (user’s piano solo, MMO orchestration only, and mixed recording). The system is designed to robustly align these despite differences in instrumentation, mixing, and presence of silences. No malicious attacker is modeled; adversarial audio or intentional deception is out of scope.

Methodology — deep read

1. Threat Model & Assumptions: The task is an offline alignment problem where the user has access to three audio types: their own piano-only recordings (Puser), a Music Minus One orchestral-accompaniment-only recording (Oacc), and a reference mixed piano and orchestra recording (Mref) from online sources. There is an assumed spectral mismatch among these recordings because each contains different instrumentations and audio mixing conditions. The adversary here is the variability introduced by acoustic differences and different performers, but there is no explicit malicious attacker. The key assumption is that the mix recording acts as a stable intermediary aligning candidate for both piano and orchestra parts.

2. Data Collection & Preprocessing: The authors recruited one pianist per concerto movement to record synchronized solo piano performances (Puser) while listening to the MMO accompaniment in headphones. MMO orchestra-only recordings (Oacc) of four classical piano concerto movements (Rachmaninov, Mozart, Beethoven, Bach) were used, along with two mix recordings per movement found on IMSLP with various quality levels. All recordings were timestamp annotated for measure downbeats to allow ground truth alignment evaluation focused on piano-active and orchestra-active segments (excluding extended silences). Data augmentation via time-scale modification (TSM) was performed on piano chunks (segments of active piano performance) using harmonic-percussive source separation to generate variants at tempo scaling factors {0.8, 0.9, 1, 1.11, 1.25}, expanding alignment scenarios. In total, 150 scenarios were generated combining piano chunks, orchestra-only recordings, and mix recordings.

3. Algorithm - Dense-Sparse DTW (DS-DTW): The approach is a three-step alignment pipeline (Fig 1):

• Align piano chunk (Pquery) to mix (Mref) using subsequence DTW with chroma features and cosine distance, leveraging the dominance of piano in Mref.
• Align orchestral accompaniment (Oacc) to mix (Mref) using the introduced DS-DTW algorithm robust to spectral mismatch.
• Combine these to infer alignment of Pquery to Oacc, allowing time-scale modification of Oacc to synchronize to Puser.

DS-DTW works by first selecting a sparse set of orchestra feature frames based on spectral flux thresholding (controlling fraction γ of features kept). This filters out frames with low flux that are spectrally ambiguous or silent. Next, a pairwise cost matrix is computed between the sparse orchestra frames and dense mix frames using cosine distance over chroma features. Dynamic programming computes a cumulative cost matrix with two modes: dense matching when sparse features are continuous, and sparse matching that allows for variable temporal gaps without penalizing intermediate frames. The algorithm enforces a maximum tempo variation factor of 2. Finally, backtracing finds the optimal alignment path, mapping sparse sparse orchestra frames to dense mix frames.

4. Training Regime: DS-DTW is parameter-free aside from the sparsity hyperparameter γ, tuned via grid search on validation splits. No trainable weights or deep models are involved except baseline comparison methods using pretrained source separation networks (Spleeter, HDemucs).

5. Evaluation Protocol: Alignment accuracy is measured by the fraction of annotated downbeats where the estimated alignment error exceeds set tolerances (0.1s, 0.2s, 0.5s, 1s, 2s). Scores are averaged across all four concertos equally to ensure no bias from recording length. Baselines include naive pairwise subsequence DTW, iterative subtractive alignment (ISA) with spectral subtraction, and source separation followed by DTW with two different pretrained models. Ablations of DS-DTW were performed by varying γ to elucidate its impact on performance.

6. Reproducibility: The authors release code, data annotations, and benchmark framework openly, excluding the MMO orchestra-only recordings due to licensing. Implementations of DS-DTW and evaluation scripts are available at the linked GitHub repository. Experimental settings and hyperparameter details are fully documented.

Example: For a piano chunk Pquery, subsequence DTW aligns it to Mref. The orchestra accompaniment Oacc is aligned against Mref using DS-DTW by first selecting high-flux frames from Oacc chroma features. DS-DTW's dynamic programming matrix is constructed between these sparse frames and all mix frames, allowing gaps in sparse sequences to handle silences or spectral mismatch regions. After computing the cost matrix and backtracing, the optimal time warping path is used with the Pquery–Mref alignment to infer the final Pquery–Oacc alignment. This alignment is then used to time-scale modify the orchestra accompaniment to synchronize with the user’s piano.

Technical innovations

• Introduction of Dense-Sparse DTW (DS-DTW), a variant of DTW that aligns a sparse set of selected audio frames against a dense reference sequence to improve robustness to spectral mismatch.
• Use of spectral flux thresholding to sparsify feature sequences, selectively focusing alignment on frames with prominent timing cues.
• Two-mode dynamic programming cost accumulation combining dense matching on consecutive sparse features and sparse matching allowing variable length gaps for better handling of rests and silences.
• Application of DS-DTW to a score-free piano concerto accompaniment alignment problem using mix recordings as intermediaries, bypassing the need for symbolic scores or complex source separation.

Datasets

• Concerto Accompaniment Benchmark — ~3 hours total — MMO orchestra-only recordings (proprietary, non-public), piano-only recordings by recruited pianists, mix recordings from IMSLP

Baselines vs proposed

• Naive Pairwise DTW: 2.7% error at 2s tolerance vs DS-DTW: 0.6%
• Iterative Subtractive Alignment: 7.5% error at 2s tolerance vs DS-DTW: 0.6%
• Source Separation + DTW (Spleeter): 4.8% error at 2s tolerance vs DS-DTW: 0.6%
• Source Separation + DTW (HDemucs): 0.3% error at 2s tolerance vs DS-DTW: 0.6%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.18189.

Fig 1: Overview of a system for customizing concerto accompaniments.

Fig 2: Visualization of DS-DTW for aligning a sparse sequence (selected

Limitations

• MMO orchestra-only recordings in the benchmark are proprietary and not publicly released, limiting full reproducibility.
• Evaluation focuses on offline alignment accuracy; no user studies or subjective listening tests reported for musical quality of accompaniment modifications.
• The approach currently assumes a maximum tempo variation factor of two, which may not cover extreme expressive timing variations.
• DS-DTW requires tuning of the sparsity hyperparameter γ; adaptive selection of γ for different recordings is left to future work.
• No explicit adversarial evaluation or testing against maliciously manipulated audio sequences.
• Mix recordings used as intermediaries vary in quality, potentially influencing alignment results unevenly.

Open questions / follow-ons

• Can DS-DTW be extended to handle online, real-time accompaniment synchronization scenarios?
• How can the sparsity parameter γ be adaptively tuned automatically per recording to optimize alignment?
• Would subjective listening tests confirm that DS-DTW’s improved alignment accuracy translates to better perceived musical synchronization?
• Can the approach be generalized to other instrumental combinations or genres beyond piano concertos?

Why it matters for bot defense

While this paper’s domain is music information retrieval rather than bot defense or CAPTCHA, the underlying challenge of aligning sequences robustly across domains with partial or noisy observations shares conceptual similarities with aligning challenging patterns in adversarial settings. The DS-DTW method’s idea of selectively focusing alignment on sparse, informative cues rather than attempting dense matching over noisy or ambiguous regions could inspire CAPTCHA or bot-detection algorithms that concentrate on stable features resistant to manipulation. Furthermore, the explicit incorporation of gap handling and flexible transitions in DTW’s dynamic programming could inform temporal alignment or user behavior modeling approaches in bot defense to tolerate natural variance while isolating suspicious deviations. However, direct application is speculative and would require domain-specific adaptations.

Cite

@article{arxiv2607_18189,
  title={ Dense-Sparse Dynamic Time Warping for Customizing Piano Concerto Accompaniments },
  author={ TJ Tsai and Kavi Dey and Yigitcan Ozer and Meinard Muller },
  journal={arXiv preprint arXiv:2607.18189},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.18189}
}

Read the full paper

• arXiv:2607.18189 (abstract)
• arXiv:2607.18189 (PDF)