On-Policy Delta Distillation

Source: arXiv:2607.15161 · Published 2026-07-16 · By Byeongho Heo, Jaehui Hwang, Sangdoo Yun, Dongyoon Han

TL;DR

This paper addresses a fundamental limitation in on-policy distillation (OPD) for reinforcement learning-based post-training of large language models (LLMs). The problem is that standard OPD directly trains a student model to imitate the teacher model’s output probabilities without explicitly capturing the reasoning improvements induced by instruction tuning. The authors propose a novel reward signal called the delta signal, defined as the difference in token-level log probabilities between the reasoning-tuned teacher and its base model prior to instruction tuning. This delta signal isolates reasoning-related knowledge changes, providing a more targeted distillation signal. The new method, On-Policy Delta Distillation (OPD2), incorporates this delta signal with carefully designed centering and joint conditioning to stabilize training.

Through extensive experiments across math, science, and code reasoning datasets, OPD2 consistently outperforms conventional OPD and a recent advanced variant (ExOPD). Gains are demonstrated across multiple model sizes (1.7B to 8B parameters) and between non-thinking and thinking inference modes. Importantly, OPD2 achieves these improvements with only 100 training steps and limited data—less than a full epoch. The results show that directly distilling the reasoning tuning trajectory via the delta signal leads to stronger knowledge transfer and more robust reasoning performance in student LLMs.

Key findings

• The delta distillation signal (difference between teacher and base models) enhances tokens linked to reasoning logic (e.g., "hence", "however"), while suppressing exploratory or conversational tokens (e.g., "try", "see", "i'm") compared to traditional OPD (Fig 2, Table 1).
• OPD2 provides consistent performance gains over OPD and ExOPD across benchmarks in math, science, and code reasoning plus differing model scales (1.7B, 4B, 8B) with improvements of 3-10+ percentage points in pass@1 accuracy (Tables 2-5).
• In non-thinking mode for Qwen3-4B, OPD2 improves average math benchmark scores from 64.0 (OPD) and 66.4 (ExOPD) to 70.3 (Table 2).
• In thinking mode, where models are already stronger, OPD2 still achieves moderate improvements (e.g., Qwen3-4B math average from 70.9 OPD to 74.8 OPD2, Table 4).
• With only 100 training steps (less than one epoch) on 100k multi-domain reasoning questions, OPD2 markedly improves reasoning without costly RL training (Sec 3.1).
• Token-level analyses show OPD2’s delta signal better suppresses probability on incorrect reasoning tokens, leading to more precise supervision than OPD (Fig 3).
• OPD2’s reward design includes centering and a sign consistency constraint to stabilize convergence and prevent overfitting to the delta signal (Eq 8).
• OPD2 outperforms the state-of-the-art ExOPD method, demonstrating the value of modeling the learning trajectory in reasoning tuning during distillation.

Methodology — deep read

1. Threat Model and Assumptions: The adversary is not explicitly modeled as this is a method paper focused on improving on-policy distillation in LLM post-training. Assumptions include availability of a teacher model with reasoning tuning and its base pre-instruction tuning model. The student model generates token sequences on-policy during training.

2. Data: The authors build a 100k sample multi-domain training set by mixing three reasoning datasets equally: OpenMathReasoning (Math), OpenScienceReasoning-2 (Science), and OpenCodeReasoning (Code). They discard original answers and generate new model completions on these questions. The datasets are publicly referenced.

3. Architecture/Algorithm: On-policy distillation (OPD) traditionally uses token-level rewards derived from the log probability difference of teacher vs student on sampled tokens. The paper innovates by defining the "delta signal" as the log probability difference between the teacher and teacher's base model prior to instruction tuning. This delta captures changes due to reasoning tuning and is used as the primary distillation reward. The new advantage function AD2_t applies centering by subtracting expected rewards and adds a sign consistency condition to avoid conflicts between delta and original OPD signals (Eq 8). Training optimizes expected reward gradients weighted by AD2_t.

4. Training Regimen: Training is conducted for 100 steps using AdamW optimizer with learning rate 5e-6 and cosine LR decay. Batch size corresponds to generating one completion per question. Softmax temperature is 0.7. KL regularization and group norm are disabled. The maximum generation length is 8k tokens. Rewards are clipped and scaled by 0.1. Training runs on GPU clusters with vLLM backend.

5. Evaluation Protocol: They evaluate on 14 benchmarks spanning math (7), science (3), and code (4) tasks. Metrics are pass@1 accuracy averaged across multiple runs (number of runs listed in paper). Comparisons are made to original baseline models, OPD, and ExOPD as an advanced baseline. Ablations on advantage functions and centering are performed to isolate impact. Statistical results and token-level analysis across 10k examples provide deeper insight.

6. Reproducibility: Code will be released at the cited GitHub repository. Models used (Qwen3, Gemma4) are open source or described in prior work. The base teacher models and instruction-tuned teachers are specified. While all datasets are public, some evaluation benchmarks and splits may not be fully open, typical for reasoning research.

Concrete example: Given a question from OpenMathReasoning, the student generates a token sequence. Both teacher and teacher base models compute token probabilities on this sequence to produce the delta reward signal R∆_t = log π_teacher - log π_base for each token. This signal is centered by subtracting the expected reward over top-k tokens of the student policy, producing A∆_t. The final advantage AD2_t is computed with sign agreement checks and used to scale the policy gradient updates for the student, enabling it to learn the reasoning improvements induced by instruction tuning. Over 100 such steps on mixed domain data, the student model progressively absorbs the reasoning capabilities encoded in this delta signal, as reflected in improved benchmark performance.

Technical innovations

• Definition and use of a delta distillation reward computed as the difference between a reasoning-tuned teacher model and its base pre-tuning model to isolate reasoning improvements.
• Integration of the delta signal into the on-policy distillation framework via a novel advantage function that centers rewards and enforces sign consistency to stabilize training.
• Extensive empirical analysis showing that the delta signal focuses supervision on reasoning-related tokens and suppresses generic or conversational tokens.
• Comprehensive multi-domain post-training setup mixing math, science, and code reasoning with short-duration on-policy distillation showing strong generalization across tasks.

Datasets

• OpenMathReasoning — 100k sampled for training — publicly available
• OpenScienceReasoning-2 — 100k sampled for training — publicly available
• OpenCodeReasoning — 100k sampled for training — publicly available
• 7 Math benchmarks (AIME24, AIME25, AMC23, HMMT25, MATH500, OlympiadBench, ReasoningGym Math) — standard evaluation sets
• 3 Science benchmarks (GPQA, SuperGPQA, SciBench) — standard evaluation sets
• 4 Code benchmarks (CodeContests, CodeForces, LiveCodeBench, ReasoningGym Algorithm) — standard evaluation sets

Baselines vs proposed

• Original Model (Qwen3-1.7B): averaged math pass@1 = 34.8 vs OPD2 = 54.6 (+19.8)
• OPD on Qwen3-4B non-thinking math: 64.0 avg vs OPD2: 70.3 avg (+6.3)
• ExOPD on Qwen3-8B non-thinking math: 67.8 avg vs OPD2: 71.6 avg (+3.8)
• Qwen3-1.7B code average pass@1 OPD: 10.5 vs OPD2 29.4 (+18.9)
• Qwen3-4B science average pass@1 ExOPD: 47.8 vs OPD2 50.5 (+2.7)
• Qwen3 thinking mode Qwen3-4B math average OPD: 70.9 vs OPD2: 74.8 (+3.9)
• OPD2 outperforms ExOPD on all tested scales and domains consistently as per extensive benchmarks in Tables 2–5

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.15161.

Fig 2: Word clouds for OPD, Base and ∆. The figure illustrates distillation signal strengths in

Fig 3: Token-level distillation signals on simple reasoning examples. Blue and red indicate

Fig 3 (page 4).

Limitations

• Experiments focus on post-training with small number of gradient steps; benefits for long-duration training remain untested.
• Although token-level analyses show promising alignment with reasoning correctness, the training and evaluation did not include stress tests against adversarially crafted reasoning failures.
• The technique requires access to both the reasoning-tuned teacher and its corresponding base pre-tuning model, which may not always be available.
• The evaluation is limited to three reasoning domains (math, science, code) and specific benchmarks; generalization to other NLP tasks or dialogue remains to be tested.
• Training relies on sample-efficient on-policy distillation, but the impact on scaling far larger models or extremely long contexts is unclear.
• The convergence stability improvements rely on heuristics (sign consistency condition); theoretical guarantees are not provided.

Open questions / follow-ons

• Could the delta signal concept be extended beyond instruction tuning to other forms of LLM fine-tuning or multi-stage training pipelines?
• How does OPD2 behave under distribution shifts or on adversarially designed reasoning tasks that exploit weaknesses in the student model?
• Can the delta distillation method be applied in zero-shot or few-shot prompting scenarios to enhance reasoning without fine-tuning?
• What are the theoretical properties of the sign consistency condition and centering operation in stabilizing policy gradients with delta signals?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this paper provides an advanced methodology for efficiently transferring complex reasoning capabilities from large teacher models to smaller student models through on-policy distillation. Applying OPD2 could improve the reasoning robustness of LLM-based challenge-response systems in CAPTCHAs without requiring computationally expensive reinforcement learning. The delta signal isolates reasoning-related knowledge from generic language features, potentially enabling more precise behavior alignment in models deployed for bot-detection or user verification tasks. Additionally, the multi-domain training approach and short post-training regime suggest scalable deployment possibilities. However, practitioners should note the need for well-aligned teacher and base models to compute the delta signal and the focus on reasoning domains which may differ from open-ended dialogue or adversarial bot scenarios.

Cite

@article{arxiv2607_15161,
  title={ On-Policy Delta Distillation },
  author={ Byeongho Heo and Jaehui Hwang and Sangdoo Yun and Dongyoon Han },
  journal={arXiv preprint arXiv:2607.15161},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.15161}
}

Read the full paper

• arXiv:2607.15161 (abstract)
• arXiv:2607.15161 (PDF)