Understanding Reasoning from Pretraining to Post-Training

Source: arXiv:2607.16097 · Published 2026-07-17 · By Jingyan Shen, Ang Li, Salman Rahman, Yifan Sun, Micah Goldblum, Matus Telgarsky et al.

TL;DR

This paper investigates the interplay between large language model (LLM) pretraining and reinforcement learning (RL) post-training in the context of complex reasoning tasks, with a focus on the chess domain as a controlled testbed. Traditional LLM studies treat pretraining and RL largely in isolation with vast uncontrolled datasets, making it difficult to attribute model improvements precisely to either stage. The authors address this by mimicking the standard LLM pipeline on chess: pretraining language models (from 5M to 1B parameters) on human chess games, supervised fine-tuning (SFT) on synthetic reasoning traces, and RL on chess puzzles with verifiable rewards. They discover a joint scaling law showing that the final post-RL performance at fixed RL compute is well predicted by the pretraining loss, and that the rate of RL performance improvement grows approximately linearly with the amount of pretraining tokens.

Beyond scaling, their analysis reveals that RL post-training doesn’t simply sharpen the SFT policy. Instead, RL amplifies moves the SFT policy already favors on easy puzzles, while on hard puzzles it surfaces correct moves that were nearly absent under SFT. They further validate this pattern transfers beyond chess by training a 1B parameter model on math-domain text, showing longer pretraining leads to higher post-RL performance and faster RL improvement. Overall, the study provides a quantitative, interpretable characterization of the pretraining-to-RL interface and presents chess as a practical, precise benchmark for studying reasoning across the full training pipeline.

Key findings

• Post-RL pass@1 performance at a fixed RL compute budget correlates strongly with pretraining validation loss, with Spearman correlation improving from -0.93 to -0.99 as RL compute increases (Fig. 3a).
• The local slope of RL reward improvements (increment per 10× RL compute) grows approximately linearly with the logarithm of pretraining tokens (Pearson r=+0.84) and model size, showing data scale drives RL gains (Fig. 3b,c).
• The compute-optimal allocation between pretraining and RL shifts towards a larger fraction of RL compute (from ~5% to ~32%) as total compute budget increases for fixed model size, indicating RL becomes more beneficial with stronger pretraining (Fig. 2).
• RL improves pass@1 substantially but has mixed or small effects on pass@16 (diversity) metrics; additional pretraining often yields better pass@16 results than more RL compute.
• Mechanistic analysis shows RL amplifies memorized correct moves on easy puzzles but discovers rare correct moves on hard puzzles, sometimes also reinforcing incorrect moves, explaining why pass@1 improves without consistent gains in pass@k (Section 4.1).
• The joint pretraining–RL scaling law combining a Chinchilla-style pretraining loss predictor with the RL slope parameter accurately predicts held-out RL trajectories (R2=0.84) and the empirical frontier of puzzle performance (Fig. 4).
• Findings generalize beyond chess to a 1B model pretrained on math-domain text, where longer pretraining also predicts higher post-RL performance and faster RL improvement slopes (Section 5).

Threat model

Not directly a security paper; the study treats the RL agent as a non-adversarial learner optimizing puzzle solving from the pretrained policy. The environment provides ground-truth rewards and verifiable correctness; there is no consideration of malicious adversaries or adversarial attacks in the RL or pretraining stages.

Methodology — deep read

The authors propose a synthetic training framework that replicates the standard LLM training stages strictly in the chess domain, enabling clean experimental control and observability.

1. Threat Model & Assumptions: The adversary is implicit here as the model learning agent, aiming to optimize puzzle solving via RL starting from pretrained policies. The study focuses on how pretraining and RL interact rather than on adversarial attacks; the environment provides verifiable ground-truth rewards based on perfect puzzle solutions.

2. Data: Pretraining uses 54 billion tokens of human chess game records from Lichess 2022, filtered by player Elo and game length to control data variability. The supervised fine-tuning uses 156,000 chess puzzles divided into five difficulty bins (B1–B5), with a separate 1,480-puzzle evaluation benchmark balanced across difficulty levels and puzzle types. Chess moves are tokenized with an 81-token vocabulary encoding piece, source, destination, and move flags.

3. Model Architecture and Training: The authors train autoregressive language models of 5M to 1B parameters following the dense Qwen3 architecture. Pretraining optimizes next-token prediction on chess move sequences. For fine-tuning, they generate synthetic reasoning traces by sampling multiple plausible continuations from the pretrained model, merging into a search-tree structure serialized as token sequences, and train the model to predict the best solution continuation. Reinforcement learning is performed on a puzzle environment that gives binary reward of 1 for exactly matching the entire ground-truth solution line, or 0 otherwise. The policy is optimized with Group Relative Policy Optimization (GRPO), adapted to this setting.

4. Training Regime: Pretraining uses up to 6.5×10^19 FLOPs with models trained on 200M to 52B tokens. RL is run for 1,000 to 5,000 steps depending on compute budget, using FLOPs as the compute measure. Seeds and hyperparameters are carefully controlled; details are in appendix (not fully specified here).

5. Evaluation Protocol: Metrics include pass@1 and pass@k (k=16) success rates on held-out puzzle benchmarks stratified by difficulty. The authors sweep over 36 combinations of pretraining and RL compute budgets across model scales to empirically measure performance frontiers. They fit scaling laws relating post-RL metrics to pretraining loss, tokens, and model size, using ordinary least squares and Spearman/Pearson correlations. Mechanistic analysis tracks policy evolution move-by-move.

6. Reproducibility: The models and datasets are publicly released on Huggingface and code on GitHub, enabling reproduction of training and evaluation. While some large compute sweeps may be costly to reproduce fully, key scaling law parameters and checkpoints are available.

Concrete Example: A 50M parameter model pretrained on Lichess games is fine-tuned with synthetic reasoning traces and then optimized with 2,000 RL steps on chess puzzles. This leads to a ~20% pass@1 downstream improvement. Pretraining loss predicts the final RL-tuned performance, and RL uncovers moves missing or rare in the SFT policy on harder puzzles, as shown in detailed puzzle traces (Fig. 7).

Technical innovations

• Use of chess as a fully controlled, verifiable testbed for studying the interaction of pretraining and reinforcement learning in reasoning models, enabling systematic compute sweeps and trace-level policy analysis.
• Introduction of synthetic reasoning traces constructed from a tree-structured merge of multiple sampled continuations from the pretrained model, enabling supervised fine-tuning that elicits model reasoning via chess move tokens without external search.
• Discovery and quantitative modeling of a joint pretraining-RL scaling law linking pretraining loss and data scale to the slope and asymptote of RL reward improvements, enabling compute-optimal allocation between pretraining and RL.
• Mechanistic analysis separating easy and hard puzzle behaviors, showing RL both amplifies pretrained policy preferences and discovers rare new correct moves, with direct policy trace verification to explain mixed empirical metrics.

Datasets

• Lichess 2022 Games — 54B tokens — public online chess dataset
• Lichess Chess Puzzles — 156K puzzles — curated subset filtered for quality and difficulty
• Chess Puzzle Benchmark — 1,480 puzzles — curated for balanced difficulty and diversity

Baselines vs proposed

• Pretrained only (no RL): pass@1 ranges from 5% to 35% depending on model size and pretraining compute; post-RL pass@1 improves this by up to 30 percentage points at large compute budgets.
• SFT without reasoning traces: improves pass@1 but not pass@16, versus SFT with synthetic reasoning traces which improves both pass@1 and pass@16 significantly.
• RL compute allocation at low total compute budgets yields better performance focusing on pretraining; at high compute budgets, increasing RL compute fraction from 5% to ~28% improves pass@1 performance.
• Post-RL policies outperform pre-RL policies on pass@1 with correlation to pretraining loss R2=0.98 at high RL compute; joint scaling law fit achieves R2=0.84 in predicting post-RL slopes of reward improvement.

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.16097.

Fig 1: Overview. (a) We introduce a synthetic framework for studying reasoning from pretraining to post-training in the

Fig 2 (page 2).

Fig 7: An example of a chess puzzle. The prompt consists of the move sequence representing the board state, optionally

Fig 21: Policy evolution on a hard puzzle (B5) across training stages. Left: board position with ground-truth move

Fig 22: Structured reasoning traces evolve during RL training. For the 20M and 50M models, panels report per-step

Limitations

• Chess is a simplified reasoning domain with compact action space and exact verifiable rewards; findings may not fully generalize to natural language reasoning with open-ended outputs or sparse/noisy rewards.
• The RL training regime covers early non-saturated parts of reward curves; global RL ceilings and long-term saturation behaviors remain unobserved.
• Synthetic reasoning traces are model-generated approximations, possibly limiting the diversity and complexity of fine-tuning compared to human explanations or provenance.
• Mechanistic analyses focus mainly on move-level policy shifts; deeper interpretability or representation-level insights are not explored.
• Evaluation metrics emphasize pass@1 correctness; metrics capturing broader aspects of reasoning diversity or sample efficiency are less studied.
• The math-domain transfer experiments are limited to a single 1B model and domain; broader cross-domain generalization needs further validation.

Open questions / follow-ons

• How do the discovered pretraining-to-RL scaling laws extend to large-scale natural language LLMs with heterogeneous, noisy datasets and more complex action spaces?
• Can explicit or learned search algorithms be combined with synthetic reasoning traces to further improve RL efficiency and policy discovery?
• What representation-level changes underlie the move-level policy shifts induced by RL, especially in surfacing rare correct moves on hard reasoning problems?
• How do different reward structures (e.g., dense, shaped rewards) during RL post-training influence the joint scaling behavior and optimal compute allocation?

Why it matters for bot defense

For bot-defense practitioners working on CAPTCHA or language-based bot detectors, this study highlights the fundamental importance of pretraining quality in enabling effective post-training improvements through reinforcement learning or policy optimization. It suggests that models initialized with stronger pretrained priors will yield greater returns from subsequent interaction-driven fine-tuning phases, thus informing optimal compute budgeting between offline data ingestion and online refinement.

Moreover, the mechanistic insights that RL can both sharpen known correct behaviors and surface rare, previously underrepresented solutions provides an analogue for CAPTCHA systems: reinforcement or continuous learning can reveal nuanced or subtle detection patterns not evident from static training alone. This work encourages practitioners to carefully evaluate how incremental training stages contribute differently to model capabilities, and to choose evaluation metrics sensitive to both common-case amplification and rare-case discovery. Lastly, the authors’ chess-based controlled pipeline offers a blueprint for building verifiable and interpretable testbeds for analyzing reasoning evolution during model updates—an approach potentially useful for robust CAPTCHA adversary modeling.

Cite

@article{arxiv2607_16097,
  title={ Understanding Reasoning from Pretraining to Post-Training },
  author={ Jingyan Shen and Ang Li and Salman Rahman and Yifan Sun and Micah Goldblum and Matus Telgarsky and Pavel Izmailov },
  journal={arXiv preprint arXiv:2607.16097},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.16097}
}

Read the full paper

• arXiv:2607.16097 (abstract)
• arXiv:2607.16097 (PDF)