UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing

Source: arXiv:2607.08646 · Published 2026-07-09 · By Xinlong Zhao, Dongsheng Liu, Hengyu Zhao, Zixuan Fu, Zheng Wang, Jie Cai et al.

TL;DR

UltraX addresses the challenge of refining large-scale pre-training corpora for Large Language Models (LLMs) as the availability of new training data saturates and scaling via more data yields diminishing returns. Existing refinement approaches fall short in simultaneously delivering fine-grained quality improvements, computational efficiency, and execution reliability at scale. UltraX proposes a novel function-calling framework that significantly expands the editing operation space to include insertion alongside deletion and modification, enabling more precise instance-level editing of text. It achieves reliable training supervision by programmatically converting expert LLM's end-to-end refined texts into structured editing operations using dataset-adaptive prompts, line alignment mapping, and dynamic context replacement. During inference, UltraX improves robustness with sliding-window prediction, global operation aggregation, and systematic post-processing to generate refined corpora with high consistency and scalability.

Empirically, UltraX pretrains 1B-parameter MiniCPM models on multiple public large corpora comprising roughly 20B tokens. Across five heterogeneous corpora and ten downstream benchmarks, UltraX consistently outperforms baseline methods including raw data and the function-calling ProX-C approach by approximately 1.53% to 2.00% relative improvement. It also attains higher final accuracy with fewer training tokens, demonstrating superior data efficiency. Ablations show the importance of instruction-guided refinement, operation distribution balancing, and combined editing functions for maximizing effectiveness. UltraX thus presents a principled, scalable, and practical solution for next-generation large-scale data refinement in LLM pretraining pipelines.

Key findings

• UltraX achieves average relative improvements of approximately 2.00% over raw data and 1.53% over ProX-C across five large corpora and ten downstream tasks (Table 2).
• UltraX attains best results on 34/50 task-corpus pairs, indicating consistent gains rather than isolated task improvements.
• UltraX reaches similar or better downstream performance with only 16B training tokens compared to 20B tokens for baselines, showing stronger data efficiency (Fig 2).
• Instruction-guided training and inference outperform no-instruction baseline by 0.41% average score increase and top-perform on 8/10 tasks (Table 3).
• Inserting insertion operation alongside deletion and modification enables finer granularity editing and prevents semantic fragmentation compared to deletion-only baselines.
• Ratio-controlled sampling by editing operation combination balances preservation and edits, improving average downstream scores over preservation-weighted sampling.
• Segment-wise sliding-window prediction with overlap-aware operation aggregation and systematic post-processing improves large-scale execution stability over prior function-calling pipelines.
• UltraX refines heterogeneous corpora including FineWeb, RedPajama-v2, AICC, Ultra-FineWeb, and FineWeb-ProX-Doc, demonstrating generality across diverse data qualities.

Threat model

The adversary is naturally occurring noise, low-quality, or inconsistent text segments in large-scale pre-training corpora. The threat consists of semantic errors, formatting noise, and structural artifacts that degrade model training quality. UltraX assumes no intelligent or adaptive adversarial entity and does not consider active attacks such as poisoning or evasion. The goal is to robustly and efficiently identify and correct erroneous content with limited trusted supervision, maximizing data utility for LLM pretraining.

Methodology — deep read

1. Threat Model and Assumptions: UltraX addresses the problem of improving large-scale pre-training corpora for LLMs by refining noisy text data to enhance downstream task performance. The adversary is implicitly the noise and low-quality content in the web-scale data. UltraX assumes access to an expert LLM to provide high-quality end-to-end refined text as seed supervision but targets efficiency and scalability by deploying lightweight refiner models in production. There is no direct adversarial threat model; instead, the focus is on data quality enhancement and scalability.

2. Data: The authors sample approximately 20 billion tokens from five large-scale pre-training corpora: FineWeb, RedPajama-v2, AICC, Ultra-FineWeb, and FineWeb-ProX-Doc. Seed data is extracted for prompt optimization and expert LLM refinement. Training, validation, and testing splits align with the original corpora. Original and expert-refined text pairs form the core data for hierarchical program supervision construction.

3. Architecture/Algorithm: UltraX models refinement as programmatic editing via function calls over structured operations — insertion (add_line), deletion (remove_lines, remove_all), and modification (replace_str). These operations are anchored by line indices and string spans to maintain clarity and executability. The refinement model is a lightweight sequence-to-sequence architecture (1B-parameter MiniCPM) fine-tuned to predict sequences of function calls from input segmented documents. Dataset-adaptive prompt optimization tunes expert LLM prompts to generate reliable seed refined texts. Hierarchical Text-to-Operation Mapping converts text diffs into structured program supervision via line alignment and dynamic context replacement to resolve ambiguities in locating edits.

4. Training Regime: Using seed supervision, the refiner is supervised with compact function call sequences representing edits. Ratio-controlled sampling balances examples with different editing operation combinations (keep, insert, delete, replace) to stabilize training distribution. System-level task instructions define operation semantics and preservation principles, improving training consistency. Training runs on standard large-scale GPU clusters. The model is trained to convergence; exact epochs and batch sizes are detailed in appendices but align with typical 1B-parameter training protocols. Random seeds and fixed instructions ensure reproducibility.

5. Evaluation Protocol: Refined corpora from UltraX and baseline methods are used to pretrain MiniCPM 1B models on identical token budgets (~20B tokens). Downstream performance is measured over ten benchmarks including ARC, CommonsenseQA, MMLU, PIQA, WinoGrande, SciQ, etc., aggregated and reported with per-task breakdowns (Table 2). Statistical significance is implied by consistent wins across tasks and corpora. Ablations analyze effects of system instructions, sampling strategy, and program function set on final performance. Cross-validation is not mentioned. Experiments include performance over varied token budgets (Fig 2).

6. Reproducibility: The authors release datasets (UltraX-Preview) and a 0.6B refinement model checkpoint on HuggingFace, along with open-source code on GitHub to reproduce the pipeline. The expert LLM refinement step depends on proprietary models but dataset-adaptive prompt optimization details are described. All critical training and inference components are documented.

Concrete Example Pipeline End-to-End: Starting with raw web corpus text, UltraX first samples seed data and probes its noise profile. Dataset-adaptive prompts are generated and fed into an expert LLM to produce an end-to-end refined text version. Original and refined texts undergo line alignment mapping to identify unchanged, deleted, inserted, or modified lines. Intra-line edits are parsed with dynamic context replacement to extract replace_str operations unambiguous in context. The resulting program supervision sequence (e.g., remove_lines(3,5), replace_str(12, 'can can', 'can'), add_line(21,0,'inserted text')) is used to supervise fine-tuning of the lightweight refiner model. During large-scale inference, documents are split via sliding windows, local operation sequences predicted, merged via global line-number space mapping and overlap-aware aggregation, then post-processed with filtering ambiguous replacements and merging overlapping edits. The finalized program is deterministically executed on original text to yield a scalable, auditable refined corpus.

Technical innovations

• Introduction of insertion operation in addition to deletion and modification to complete the editing function space for fine-grained instance-level data refinement.
• Development of a reliable program-supervision generation pipeline combining dataset-adaptive prompt optimized expert LLM refinement, line alignment mapping, and dynamic context replacement to convert text diffs into structured program supervision.
• Design of a large-scale inference pipeline using sliding-window prediction, global operation aggregation, and systematic post-processing (including ambiguous replacement filtering and consecutive operation merging) to improve robustness and execution reliability at scale.
• Use of ratio-controlled sampling of operation combinations and low-confidence example filtering to stabilize training data distribution and improve model generalization.

Datasets

• FineWeb — ~20B tokens — public web corpus (Penedo et al., 2024)
• RedPajama-v2 — ~20B tokens — public pretraining corpus (Weber et al., 2025)
• AICC — ~20B tokens — academic pretraining corpus (Ma et al., 2025)
• Ultra-FineWeb — ~20B tokens — curated web corpus (Wang et al., 2025)
• FineWeb-ProX-Doc — ~20B tokens — derivation of FineWeb with ProX refinement (Zhou et al., 2024)

Baselines vs proposed

• Raw data pretraining: average downstream score = 45.08 (FineWeb corpus)
• ProX-C (function-calling deletion and modification model): average score = 45.05 (FineWeb)
• UltraX (insertion, deletion, modification): average score = 46.14 (FineWeb), +2.0% relative over Raw, +1.53% over ProX-C
• At 16B tokens, UltraX surpasses Raw 20B-final score (45.49 vs 45.08), showing higher data efficiency
• Instruction-guided UltraX improves average score by 0.41% over non-instruction variant and scores best on 8/10 tasks

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.08646.

Fig 1: Overall workflow of UltraX, covering program-supervision generation, refinement model training,

Fig 2 (page 4).

Fig 3 (page 4).

Limitations

• Dependence on expert LLMs to generate high-quality end-to-end refined text supervision may limit applicability if such models are unavailable or costly.
• Evaluation focuses on downstream benchmarks post-pretraining; the impact on robustness to adversarial or distribution-shifted noise is not explicitly studied.
• The refinement model is a 1B-parameter MiniCPM architecture; model scalability to larger sizes, or alternate architectures, is unexplored.
• While UltraX improves data quality, the approach assumes clean line segmentation and explicit line numbering, which may limit generality to more unstructured corpora.
• Seed supervision quality depends on alignment and context replacement heuristics, which may introduce systematic biases or miss some complex edits.
• The proposed editing program space, though more complete, still relies on line anchoring and may struggle with cross-line semantic changes.

Open questions / follow-ons

• Can UltraX scale to larger model sizes and more highly heterogeneous or multimodal corpora beyond text-only datasets?
• How robust is UltraX to adversarial data noise or active poisoning attacks, and can the program supervision pipeline be adapted for defense?
• Can the function space be extended to support more complex semantic restructuring beyond line-based edits, such as paragraph-level reordering or content synthesis?
• How does UltraX refinement impact specialized tasks such as few-shot learning or domain adaptation when integrated into large LLM training?

Why it matters for bot defense

UltraX's approach highlights the importance of precise, fine-grained programmatic editing of large-scale corpora to improve training data quality for language models. For bot-defense or CAPTCHA practitioners, the methodology of converting raw noisy text into auditable, structured refinement operations via function calls can inspire similar structured data correction pipelines in web content curation, user-generated input sanitization, or anomaly detection. The sliding-window prediction and overlap-aware aggregation techniques may also inform scalable processing of long or streaming textual inputs in real-time bot detection systems.

Moreover, UltraX demonstrates that combining expert LLM guidance with lightweight refinement models can reconcile quality and efficiency trade-offs — a principle applicable in designing CAPTCHA challenge generation or filtering pipelines where semantic coherence and operational speed both matter. The concept of defining explicit operation spaces (insertion, deletion, replacement) tied to structured positions rather than opaque text generations could enable interpretable and auditable bot-defense filters, minimizing hallucinations or unintended side effects common in purely generative approaches.

Cite

@article{arxiv2607_08646,
  title={ UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing },
  author={ Xinlong Zhao and Dongsheng Liu and Hengyu Zhao and Zixuan Fu and Zheng Wang and Jie Cai and Jie Zhou and Qiang Ma and Xuanhe Zhou and Xu Han and Yudong Wang and Zhiyuan Liu },
  journal={arXiv preprint arXiv:2607.08646},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.08646}
}

Read the full paper

• arXiv:2607.08646 (abstract)
• arXiv:2607.08646 (PDF)