MultiSynt/MT: Trillion-Token Multi-Parallel Pre-Training Data Translated Across 36 Languages

Source: arXiv:2607.00890 · Published 2026-07-01 · By Maximilian Idahl, Jörg Tiedemann, Sampo Pyysalo, David Salinas, Tomasz Galica, Shenbin Qian et al.

TL;DR

MultiSynt/MT is an open synthetic parallel corpus created by translating approximately 100 billion high-quality English tokens from Nemotron-CC into 36 European languages, resulting in around 4.8 trillion target-language tokens. The corpus is constructed using a combination of large LLM-based translation systems (TOWER+ 9B and 72B) and smaller neural MT systems (OPUS-MT and HPLT-MT). This yields the largest publicly available pre-training dataset for many medium- and lower-resource European languages, surpassing existing native data baselines by orders of magnitude in volume for the lowest-resource cases. Reference multilingual language models trained on MultiSynt/MT closely match or outperform native-data baselines (HPLT 2.0) on a broad multilingual benchmark suite, achieving the native baseline's score using roughly 72% fewer training tokens and outperforming it by approximately 15% relative when training budgets are matched at 100B tokens.

In-depth analyses reveal that standard multiple-choice multilingual benchmarks miss translation-quality differences between models trained on different MT systems, which a fluency-sensitive LLM-as-judge evaluation recovers. The findings also show that Norwegian idiomatic and culturally grounded tasks remain better served by native data, while tasks requiring commonsense reasoning benefit similarly or better from translated corpora. The corpus and aligned translations from multiple MT systems are released openly to enable controlled research on the influence of translation system choice, data scaling, and native versus synthetic pre-training data quality on multilingual LLM benchmarks and capabilities.

Key findings

• MultiSynt/MT provides approximately 4.8 trillion target-language tokens across 36 languages, exceeding HPLT 3.0 native data by more than an order of magnitude on many lower-resource European languages (e.g. Maltese 182×, Irish 153× token volume).
• Reference multilingual LLMs trained on MultiSynt/MT reach the HPLT 2.0 native data baseline score with roughly 28B training tokens, about 72% fewer than the native baseline's token budget.
• At a matched 100B-token training budget, models trained on MultiSynt/MT outperform the native HPLT 2.0 baseline by approximately 15% relative on an averaged multilingual benchmark suite covering 5 languages.
• Differences in MT system quality (TOWER+ 72B > TOWER+ 9B > OPUS-MT) are not captured by standard multiple-choice benchmarks but are revealed by a fluency-sensitive LLM-as-judge evaluation using free-form generations.
• Models trained on translated data (MultiSynt/MT) have embedding neighborhoods more aligned with benchmark documents than those trained on native data, indicating benchmark alignment bias toward translated corpora.
• On Norwegian idiomatic expression tasks, native-data-trained models outperform translated-data models consistently, showing translationese limitations on idiomatic and culturally grounded knowledge.
• On Norwegian commonsense reasoning tasks, translated-data models close the performance gap and match or slightly exceed native-data models at larger training budgets, indicating preservation of transferable reasoning patterns in translated text.
• Translation-based corpus quality gains may stem primarily from the high quality of the English source corpus (Nemotron-CC HQ) rather than translation alone, as the comparison with native data also varies source corpus distribution.

Threat model

n/a - This work does not focus on adversarial threat models but rather addresses the resource and quality limitations of multilingual pretraining data by producing a large-scale synthetic parallel corpus via machine translation.

Methodology — deep read

The authors address the multilingual data scarcity problem by building MultiSynt/MT, a massive multi-parallel pre-training corpus created by translating a 100-billion-token high-quality English web corpus, Nemotron-CC HQ, into 36 European languages.

1. Threat Model & Assumptions: The focus is on producing large-scale multilingual pre-training data for low- and medium-resource languages without addressing adversarial settings. The main concern is the potential artifacts and cultural mismatch introduced by synthetic translation data rather than malicious attacks.

2. Data Collection and Preprocessing: The source corpus is a uniform random sample of approximately 155 million documents from Nemotron-CC HQ, which is quality-filtered Common Crawl English web text. This source is chosen for its strong downstream performance. Documents (rather than isolated sentences) are translated maintaining alignment.

3. Translation System Pool: Three translation systems are used: (a) TOWER+ 9B and 72B, large LLM-based translation-specialized models for mostly higher-resource European languages, and (b) OPUS-MT and HPLT-MT, smaller, classical NMT models covering 36 languages including low-resource ones. A small human rating study on seven languages rates TOWER+ as highest quality.

4. Translation Pipeline: Two parallel pipelines translate the documents with these systems. The LLM-based translation runs on a large HPC cluster with NVIDIA A100 GPUs using a vLLM orchestration, consuming about 3.1 million GPU-hours total. The NMT pipeline uses Marian-NMT on an AMD cluster with sentence-splitting and document reassembly to preserve alignments.

5. Corpus Composition: The corpus contains approximately 4.8 trillion tokens in target languages, with a total of 7.1 trillion tokens across all translation variants. The data is released with row-aligned splits allowing cross-system and cross-language controlled analyses.

6. Model Architecture and Training: Reference language models are dense LLaMA-like transformers with 24 layers and 2.15 billion parameters total (including embeddings). The same architecture and optimizer settings are used for all models, isolating the effect of training data. Models are trained on a 100B-token budget using Megatron-LM on the LUMI supercomputer with 64 AMD GPUs total. Training uses Adam with typical transformer hyperparameters.

7. Evaluation: Downstream evaluation uses multiple open datasets spanning languages dan, nld, ita, por, swe, among others. Benchmarks include multiple-choice question answering and reasoning tasks aggregated into a mean score. Additional analyses use an LLM judge to score free-text fluency differences, and embedding space analysis to study benchmark alignment. Norwegian-specific idiomatic and commonsense datasets are used to study culturally grounded knowledge.

8. Reproducibility: The full corpus and translation code are released openly, including Parquet format files preserving alignment and metadata. The MT systems are open-source or public models, and evaluation scripts are based on existing open frameworks LightEval and LM-evaluation-harness. Training hyperparameters and infrastructure details are fully documented in appendices.

Example end-to-end: The English Nemotron-CC HQ documents are randomly sampled, translated into Swedish using TOWER+ 9B on the HPC cluster, preserving document-level alignment. The resulting Swedish tokenized corpus is used to pre-train the reference 2.15B parameter LLaMa-like model on 100B tokens. The trained Swedish model is evaluated on Swedish benchmarks periodically, showing superior performance compared to a native Swedish HPLT 2.0 baseline model trained on 72% more tokens.

Technical innovations

• A trillion-token-scale multi-parallel multilingual corpus produced by large-scale high-quality synthetic translation of a 100B-token English benchmark source into 36 European languages.
• Combining LLM-based and classical NMT translation systems for broad language coverage and large-throughput production-scale corpus generation, with multiple translations per language for controlled system comparison.
• Using row-aligned multi-way document-level parallelism to enable cross-language and cross-system controlled experiments, preserving document identity across 155 million documents.
• Applying an LLM-as-judge fluency evaluation that reveals quality differences between synthetic translation systems that standard multiple-choice benchmarks fail to detect.

Datasets

• Nemotron-CC HQ — 100 billion tokens — quality-filtered English web data (Common Crawl-based)
• MultiSynt/MT — approximately 4.8 trillion target-language tokens — produced by translation of Nemotron-CC HQ across 36 European languages (openly released)
• HPLT 2.0 — native multilingual baseline dataset — publicly available (Oepen et al., 2025)
• NorEval — Norwegian native-authored idiomatic and commonsense tasks — publicly released native-authored benchmark suite

Baselines vs proposed

• HPLT 2.0 (native baseline): average multilingual benchmark score at 100B training tokens = baseline
• MultiSynt/MT (OPUS-MT translation): reaches HPLT 2.0 baseline score at ~28B tokens (72% fewer tokens), outperforms HPLT 2.0 by ~15% relative at 100B tokens
• MultiSynt/MT (TOWER+ 9B): matches or slightly exceeds OPUS-MT variant performance on Swedish benchmark
• MultiSynt/MT (TOWER+ 72B): generally highest fluency judged translation variant, slightly outperforming 9B and OPUS-MT variants but with small downstream differences

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.00890.

Fig 7: Controlled training experiment on the Nemotron-CC source data, with and without decontamination. A

Limitations

• Comparison of MultiSynt/MT to native HPLT 2.0 baseline varies both source corpus and translation; attribution of gains to translation vs. better source corpus quality is unresolved.
• Human MT translation quality evaluation covers only seven languages with a single annotator per language, limiting robustness of conclusions about translation system ranking.
• Evaluation focuses on European languages and a single 1.7B-parameter model scale; results may not generalize to lower-resource, non-European languages or larger model scales.
• Only single-seed training runs reported; no statistical variance or multiple seeds to assess stability of reported gains.
• Embedding-space analysis of benchmark alignment is descriptive and correlational, not demonstrating causation for observed performance improvements.
• Standard multiple-choice benchmarks used in main evaluation are translation-derived and biased toward translated data, potentially inflating MultiSynt/MT performance relative to native data.
• Translationese artifacts and cultural anchoring gaps remain inherent in translated corpora, limiting performance on idiomatic and culturally grounded tasks (e.g., Norwegian idioms).
• Releasing a large translated corpus risks recursive synthetic data contamination in future MT system training, potentially causing long-tail erosion effects over time.

Open questions / follow-ons

• How much of the observed downstream performance gain derives from the higher quality of the Nemotron-CC English source data versus the translation step itself?
• How do these findings generalize to lower-resource and non-European languages included in the MultiSynt/MT corpus but not evaluated here?
• What are the effects of scaling up model size beyond 1.7B parameters on the comparative performance of native versus translated pre-training data?
• Can native and synthetic translated data be optimally combined or adapted during pre-training to mitigate idiomatic and cultural knowledge gaps?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, MultiSynt/MT exemplifies a practical approach to massively scaling multilingual pre-training with synthetic data when native data are scarcer or unavailable, especially for lower-resource European languages. The careful analyses showing that standard multiple-choice benchmarks can mask translation quality differences underline the need to use multiple evaluation modalities, including fluency-sensitive metrics, to fairly assess language model capabilities across languages. The persistent gap for idiomatic and culturally grounded tasks suggests that relying solely on synthetic translation corpora may leave vulnerabilities or blind spots in systems requiring nuanced language understanding tied to local culture or idiom. Practitioners designing multilingual bot detection or challenge-response systems can take from this the importance of incorporating native or culturally grounded data where feasible, and using rigorous evaluations that go beyond standard benchmark selection accuracy to probe model fluency and naturalness. The release of MultiSynt/MT with aligned multiple translation variants also provides a valuable resource for research into robustness and sensitivity of multilingual language models to translation artifacts that can be exploited or cause failure modes in real-world deployed systems.

Cite

@article{arxiv2607_00890,
  title={ MultiSynt/MT: Trillion-Token Multi-Parallel Pre-Training Data Translated Across 36 Languages },
  author={ Maximilian Idahl and Jörg Tiedemann and Sampo Pyysalo and David Salinas and Tomasz Galica and Shenbin Qian and Tudor Nicolae Mateiu and Zihao Li and Anna Lokrantz and Fedor Vitiugin and André F. T. Martins and Jenna Kanerva and Filip Ginter and Matthias Lindemann and Tim Isbister and Birger Moell and Jonas Lindh and Jan Hajič and Jenia Jitsev and Andrey Kutuzov and Stephan Oepen and Gema Ramírez-Sánchez },
  journal={arXiv preprint arXiv:2607.00890},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.00890}
}

Read the full paper

• arXiv:2607.00890 (abstract)
• arXiv:2607.00890 (PDF)