MulTTiPop: A Multitrack Transcription Dataset for Pop Music

Source: arXiv:2607.08756 · Published 2026-07-09 · By Nathan Pruyne, Benjamin Stoler, William Chen, Chien-yu Huang, Shinji Watanabe, Chris Donahue

TL;DR

MulTTiPop addresses a critical gap in automatic music transcription (AMT) research by providing a curated dataset of multitrack MIDI transcriptions aligned to real-world commercially produced pop music audio. Unlike prior datasets focused on solo instruments, classical music, or synthetic audio, MulTTiPop offers 572 segments (3.5 hours) drawn from diverse popular music genres spanning from the 1930s to 2000s. The dataset aligns multitrack MIDI files from the Lakh MIDI Dataset with audio segments sourced from YouTube via TheoryTab metadata. This involved a novel beat-based alignment method combined with human annotation to select precise anchor beats for time-warping MIDI to audio timing. The authors evaluate state-of-the-art AMT models (MT3 and YourMT3+) on MulTTiPop, revealing substantial room for improvement, as the best model achieves only 38% Onset F1, underscoring the dataset's difficulty and real-world relevance.

Key findings

• MulTTiPop comprises 572 pop music segments totaling 3.5 hours of aligned multitrack MIDI and commercial audio from 374 songs and 263 artists.
• The anchor beat alignment process successfully produced valid alignments for 49.1% of segments; failures were mainly due to metadata mismatches (12.2%) or alignment issues.
• Melody matching in the anchor beat selection raised successful alignments from 2.3% (base metric) to 31.7%, and combined YouTube video timing plus melody increased that to 40.0%.
• Compared to existing datasets, MulTTiPop features over 100 music genres vs RWC-Pop's 2 genres, despite a smaller dataset size.
• Onset F1 for AMT models MT3 and YourMT3+ on MulTTiPop ranged from 25.29% to 38.83%, much lower than performance reported on synthesized or classical datasets like MusicNet.
• YourMT3+ outperformed MT3 slightly in harmonic-percussive transcription but both struggled with rhythm accuracy and multi-instrument consistency.
• MulTTiPop segments average 22 seconds with roughly 583 MIDI notes each, providing dense, realistic transcription challenges.
• Dataset split into dev (169 segments, 61 min) and test (403 segments, 152 min) with no artist overlap; intended only for evaluation, not training.

Threat model

n/a — The paper is not a security paper but rather presents a dataset and evaluation benchmark for music transcription. The implicit adversarial challenge is that current AMT models cannot reliably transcribe multitrack pop music from commercial audio, which is a realistic deployment scenario.

Methodology — deep read

The authors set out to create a multitrack transcription dataset aligned to commercial pop music audio to better evaluate AMT models in real-world conditions. The threat model is implicit: AMT systems transcribe commercial pop audio; the dataset is meant for evaluation only, avoiding issues from copyrighted training data.

Data sources included 25,947 segments from TheoryTab, scraped YouTube audio with user-provided melody/chord annotations, and 31,305 multitrack MIDI files from the Lakh MIDI Dataset matched to the Million Song Dataset metadata. They performed fuzzy metadata matching using Levenshtein distance on song title and artist between TheoryTab audio segments and Lakh MIDI to generate 1,164 candidate matches.

To time-align the MIDI and audio, they first obtained beat positions in the audio using an RNN-based beat tracker (madmom library) on padded audio segments. MIDI beats came from the MIDI file’s beat grid. Because the MIDI tempo may differ from audio or vary across the segment, alignment required warping MIDI note timings by linear interpolation between the matched audio beat times and candidate MIDI beats.

Anchor beat selection (defining which MIDI beat aligns to the first audio beat) is critical. They generated up to 4 candidate anchor beats per segment using various similarity metrics: chroma and onset envelope correlation between warped MIDI and audio (base method), melody matching to TheoryTab’s melody annotation, YouTube video timing constraints, and combinations thereof. Human annotators then selected the best anchor beat among candidates by listening and visually inspecting synthesized MIDI overlays against original audio. Annotators rejected unmatched pairs, flagging metadata or alignment issues.

Six expert annotators from CMU individually reviewed 9 dataset chunks (~120 samples each), with quality control via control sets embedded to ensure consistency. Final validated alignments were exported as time-warped MIDI files for transcription evaluation.

The resulting dataset contains 572 validated segments, totaling 3.5 hours of audio, from 263 unique artists and 374 unique songs, spanning Western popular music genres and decades ranging 1939-2009. Segments average 22 seconds and 583 MIDI notes. The dataset was split 70/30 into dev and test sets stratified by artist to avoid overlap. No training split was created, as the dataset is intended solely as an evaluation benchmark.

AMT evaluation used two state-of-the-art models: MT3 and YourMT3+. The models were evaluated on MulTTiPop with standard 50ms onset tolerance and two instrument label granularities: exact instrument matching and harmonic-percussive grouping. Onset F1 performance was substantially below existing benchmarks like MusicNet or Slakh2100, exposing the difficulty of real-world commercial pop transcription. Qualitative analysis revealed the models often failed to maintain consistent multitrack transcriptions or rhythmic nuance.

No code or weights release is explicitly stated. The authors provide a web-based demo for dataset exploration. The human-in-the-loop anchor beat selection and expert annotations are the core reproducibility mechanism. Metadata linking source YouTube videos enables others to retrieve the original audio segments for evaluation.

Technical innovations

• Integration of beat tracking-based fine-grained time-warping to align multitrack MIDI tempo grids with real-world audio tempo fluctuations.
• Candidate anchor beat generation combining harmonic chroma similarity, onset correlation, melody matching, and YouTube video timing metadata to identify correct MIDI-audio alignment points.
• Human-in-the-loop annotation interface allowing selection and validation of optimal anchor beats among automatically generated candidates.
• Construction of a curated multitrack pop music transcription dataset with real commercial audio, enabling evaluation of AMT models in scenarios closer to real deployment.

Datasets

• MulTTiPop — 3.5 hours — aligned commercial pop audio with multitrack MIDI
• Lakh MIDI Dataset (subset) — 31,305 songs — public dataset of multitrack MIDI
• TheoryTab — 25,947 audio segments — user annotated commercial pop audio on YouTube
• RWC-Pop — 7 hours — original pop MIDI recordings

Baselines vs proposed

• MT3 exact instrumentation Onset F1 = 28.42% vs MulTTiPop state-of-the-art ~38% ceiling for best models
• YourMT3+ exact instrumentation Onset F1 = 25.29%
• MT3 harmonic-percussive Onset F1 = 36.83%
• YourMT3+ harmonic-percussive Onset F1 = 37.87%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.08756.

Fig 1: MulTTiPop contains segments of YouTube audio beat-

Fig 2: The methodology of creating MulTTiPop. We per-

Fig 3 (page 2).

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 2).

Fig 3: Interface for annotators to select anchor beats among

Fig 4: Visualization of genres present in MulTTiPop

Limitations

• Alignment succeeds for only approximately half (49.1%) of candidate audio-MIDI segment pairs; remaining failures due to metadata or tempo mismatch.
• Dataset size is small (3.5 hours, 572 segments), limiting suitability for training complex models.
• Dataset is Western-pop biased in genre, artist, and musical style—less representative of global music styles.
• No training split or validation set provided, dataset intended solely for evaluation.
• Models evaluated were not fine-tuned on MulTTiPop, so reported performance may not reflect potential improvements with tailored training.
• Human annotation introduces subjective judgments; though quality controls exist, no inter-annotator agreement statistics reported.

Open questions / follow-ons

• Can AMT models be effectively fine-tuned or pre-trained on datasets like MulTTiPop (or similarly curated commercial audio) to improve real-world transcription performance?
• How can automatic anchor beat alignment and tempo warping algorithms be improved to reduce reliance on human annotation?
• What augmentations or architectural innovations in AMT models can better capture multitrack texture and rhythm complexity seen in commercial pop music?
• How does transcription performance vary across subgenres, decades, and instrumentation diversity within MulTTiPop?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners working on audio-based challenges or automated media processing, MulTTiPop exemplifies the complexities of real-world multitrack audio transcription outside synthetic or narrowly defined instrument datasets. The dataset highlights that state-of-the-art transcription models struggle with authentic pop music recordings, underscoring the difficulty of reliable note-level symbolic extraction in complex auditory scenes. Techniques like beat-based alignment, combined automated and human validation, illustrate best practices for generating high-quality ground truth from noisy commercial sources. While directly unrelated to bot authentication, the methodologies for robust audio-symbolic alignment and evaluation metrics may inform audio challenge design, detection of synthesized versus live audio, or adversarial robustness analyses involving complex soundscapes. Understanding the limitations and dataset biases is critical when considering real-world applications of AMT beyond controlled conditions.

Cite

@article{arxiv2607_08756,
  title={ MulTTiPop: A Multitrack Transcription Dataset for Pop Music },
  author={ Nathan Pruyne and Benjamin Stoler and William Chen and Chien-yu Huang and Shinji Watanabe and Chris Donahue },
  journal={arXiv preprint arXiv:2607.08756},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.08756}
}

Read the full paper

• arXiv:2607.08756 (abstract)
• arXiv:2607.08756 (PDF)