Reasoning Core: Designing Broad Procedural Data for Completion-Supervised Reasoning Training
Source: arXiv:2608.05148 · Published 2026-08-05 · By Damien Sileo, Valentin Lacombe, Dimitri Kachler
TL;DR
This paper introduces Reasoning Core, a large, diverse collection of 50 procedurally-generated reasoning problem environments designed specifically for completion-supervised fine-tuning (SFT) of language models. Unlike previous work focused primarily on reinforcement learning (RL) environments, Reasoning Core targets supervised learning with explicit semantic scoring, difficulty calibration, and compact answer formats to maximize training utility. The authors compare Reasoning Core with three prior procedural collections—Procedural Warmup, Reasoning Gym, and SynLogic—under a matched training protocol across four base models ranging from 135M to 3B parameters. They find that Reasoning Core achieves the strongest aggregate improvements on held-out reasoning benchmarks including DROP, LogiQA, and ARC-Challenge. Task-level analyses reveal that semantic validity alone is insufficient for training effectiveness; well-calibrated difficulty and compact canonical answer formats significantly boost utility. Extensive audits combining model-assisted diagnostics, human adjudication, and regression testing identify subtle correctness flaws in other collections, illustrating that procedural generation does not guarantee quality without careful validation. The collection, generated datasets, and validation tools are publicly available to facilitate future research.
Key findings
- Reasoning Core improves held-out reasoning NLL by up to 15–20% relative to main-only baseline across model scales (Fig 1).
- At 3B scale and 2400 updates, Reasoning Core achieves DROP F1 of 41.7% vs 33.1% main-only and is above Reasoning Gym (39.2%) and SynLogic (32.5%) (Table 2).
- Canonical, compact answer formats outperform step-by-step rationale targets in supervised fine-tuning, improving transfer to reasoning tasks.
- Difficulty controls and calibrated task complexity active up to level 5 keep tasks learnable and computationally efficient.
- Semantic audits find non-trivial default-path logical scoring bugs in 13/105 Reasoning Gym tasks and 9 SynLogic tasks, exposing critical mismatches between generation and scoring.
- Procedural data consumption fraction set at 20% prompt+answer tokens balances reasoning gains without degrading retention on broad instruction data.
- Task-level isolated interventions correlate moderately (~0.68 Kendall’s tau) across model scales, allowing developmental tuning with smaller models.
- Zero-shot performance remains below 0.75 reward on 18/50 tasks even for 300B+ parameter models, indicating sustained headroom for procedural fine-tuning.
Threat model
n/a — the paper addresses data design and training for reasoning capability in language models, not adversarial security threat models.
Methodology — deep read
The paper's core methodology addresses how to design and validate a broad collection of procedural reasoning problems for completion-supervised fine-tuning (SFT) of large language models.
Threat model & assumptions: The adversary in this context is not a security threat but rather the training setting where the model underfits or underperforms on diverse reasoning problems if training data is poorly chosen. The fundamental assumption is that carefully designed procedural problems with verifiable targets substantially improve reasoning capabilities in SFT.
Data provenance, size, and labels: Reasoning Core consists of 50 distinct procedural generators spanning nine domains (mathematics, formal proof, logic, planning/games, state tracking, graphs/constraints, formal languages, sets/structured data, and code). Each generator produces input prompts and deterministic, semantically verified answers via integrated solvers. Tasks include difficulty controls and validate semantics with execution, symbolic equivalence, or logical entailment checks. The authors generated roughly 10B tokens of data publicly released with detailed manifests and audit artifacts.
Architecture/algorithm: The paper does not introduce a new model architecture but focuses on procedural data design for existing LM architectures. Training uses standard autoregressive completion supervision with deterministic canonical answer serialization designed to reduce randomness and improve learnability. Semantic scoring functions accept multiple semantically valid outputs during evaluation, allowing flexible correctness beyond exact string match.
Training regime: Experiments use four base pretrained models (SmolLM 135M, 360M, OLMo 1B, and SmolLM3 3B) with a shared optimization setup and token budget. Auxiliary procedural data replaces 20% of prompt+answer tokens in the training mixture. Maximum sequence length is 1024 tokens. Training ranges from 300 to 2400 updates, with batch sizes and learning rates fixed across all collections. Seeds synchronize data order for paired comparisons.
Evaluation: Held-out zero-shot evaluation uses a mixed battery of downstream reasoning benchmarks: DROP, LogiQA, ARC-Challenge, and BBH-test as a combined metric assessing reasoning transfer, and MMLU-other plus DOLCI to measure retention of broader capabilities. Metrics are mostly relative reductions in held-out answer negative log-likelihood (NLL), supplemented by accuracy and F1 at the 3B scale. Task-level ablations analyze learning curves, native reward progression, and answer formats. Free-generation evaluations probe zero-shot predictive power for up to 1.6T parameter models. Additionally, a single-seed verifier-backed RL training comparison is presented.
Reproducibility: The Reasoning Core library, generated datasets (approximately 10 billion tokens), manifests, behavior caches, and audit materials are made publicly available. Dockerized external solvers and solvers with pinned dependencies ensure consistent semantic scoring across machines. The codebase exposes a unified interface for data generation, scoring, and evaluation, supporting community validation and extension.
Concrete example: For instance, a generator produces a multi-hop logical inference problem with a set of evidence and a query. It applies a canonical ordering to select a deterministic target answer (e.g., entailment classification as YES/NO). Difficulty is controlled by the number of inference steps or evidence items. Semantic scoring verifies that predicted entailments match logical entailment relations. This example task exposes the model to a broad hypothesis space, a clean training target with reduced answer ambiguity, and difficulty calibrated for learnability, contributing to downstream performance improvements shown in benchmark datasets.
Technical innovations
- A large, heterogeneous collection of 50 procedural generators spanning diverse reasoning domains tailored for completion-supervised fine-tuning.
- Design of compact canonical answer formats combined with semantic scoring that tolerates equivalent alternative answers but enforces deterministic supervised targets.
- Integrated difficulty controls mapping scalar levels to generator parameters, enabling calibrated, learnable complexity up to practical computational limits.
- A novel auditing framework combining model-assisted generation validation, human adjudication, and regression testing exposing subtle generation-to-scoring inconsistencies.
Datasets
- Reasoning Core — ~10 billion tokens — publicly released procedural generators and datasets
- Procedural Warmup — size unspecified — publicly available procedural tasks
- Reasoning Gym — unspecified large collection — external procedural RL environments
- SynLogic — unspecified size — external logical games environments
- FineWeb-Edu and DOLCI — curated educational and instruction datasets used as main training data
Baselines vs proposed
- Main-only baseline (SmolLM3-3B-Base at 2400 updates): DROP F1 = 33.1%, LogiQA Acc = 46.8%, ARC-Challenge Acc = 50.7%, BBH-test = 43.6%.
- Reasoning Core (same settings): DROP F1 = 41.7%, LogiQA Acc = 47.8%, ARC-Challenge Acc = 51.3%, BBH-test = 45.3%.
- Reasoning Gym: DROP F1 = 39.2%, LogiQA Acc = 46.7%, ARC-Challenge Acc = 51.1%, BBH-test = 40.7%.
- SynLogic: DROP F1 = 32.5%, LogiQA Acc = 47.1%, ARC-Challenge Acc = 49.7%, BBH-test = 41.6%.
- Procedural Warmup: DROP F1 = 31.3%, LogiQA Acc = 47.7%, ARC-Challenge Acc = 49.1%, BBH-test = 42.3%.
- In verifier-backed RL with Qwen2.5-3B-Instruct, Reasoning Core achieves 38.6% BBH-test accuracy vs 23.1% with Reasoning Gym.
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2608.05148.
Fig 1 (page 1).
Limitations
- All experiments are capped at 3B parameter models; scaling behavior at larger models remains unexplored.
- Evaluation focuses on formal, closed-answer reasoning; impact on open-ended, multimodal, or agentic tasks is unknown.
- Does not compare to matched mixtures of curated human-labeled datasets such as GSM8K or MATH for reasoning supervision.
- RL experiments are limited, single-seed runs that illustrate feasibility but do not provide robust statistical conclusions.
- No experiments investigating sequential use of procedural SFT followed by RL with the same collections.
- Model-family coverage is limited to a few small to medium transformer architectures, limiting generality.
Open questions / follow-ons
- How do these procedural collections perform as pretraining or combined SFT+RL curricula at large model scales (>10B parameters)?
- Can insight into compact canonical targets inform improved answer representations for open-ended or natural language reasoning tasks?
- What are optimal procedures for blending procedural with human-curated instruction data to balance reasoning gains and capability retention?
- Can the auditing methods developed here be automated further to guarantee high procedural data correctness at scale?
Why it matters for bot defense
For bot-defense and CAPTCHA practitioners aiming to improve challenge quality and robustness using procedural generation, this work offers valuable principles and tooling. It demonstrates that broad, diverse procedural data with calibrated difficulty and semantically validated compact answers can effectively train language models on complex reasoning tasks. The auditing framework highlights risks of silent correctness errors in procedural generators, emphasizing the need for thorough validation when deploying automated data generation for security challenges. The tight coupling between semantic validity and training utility suggests CAPTCHA designers should prioritize canonical, compact challenge answers with verifiable scoring rather than producing overly verbose or ambiguous targets. Moreover, the methodology for difficulty control can help generate challenges across skill levels, supporting adaptive or progressive verification schemes in bot defense. Overall, this research complements CAPTCHA engineering by providing validated design templates and a diagnostic methodology to enhance the reliability and learning benefit of procedurally-generated reasoning test data.
Cite
@article{arxiv2608_05148,
title={ Reasoning Core: Designing Broad Procedural Data for Completion-Supervised Reasoning Training },
author={ Damien Sileo and Valentin Lacombe and Dimitri Kachler },
journal={arXiv preprint arXiv:2608.05148},
year={ 2026 },
url={https://arxiv.org/abs/2608.05148}
}