DemoPSD: Disagreement-Modulated Policy Self-Distillation

Source: arXiv:2607.02502 · Published 2026-07-02 · By Yunhe Li, Hao Shi, Wenhao Liu, Mengzhe Ruan, Hanxu Hou, Zhongxiang Dai et al.

TL;DR

This paper addresses a critical challenge in on-policy self-distillation (OPSD) for large language models: privileged information leakage. In OPSD, a single model acts as both teacher and student, where the teacher has access to privileged information (e.g., verified reasoning traces or ground truth) unavailable to the student at test time. Directly forcing the student to imitate the teacher's token-level dense supervision causes the student to overfit answer-dependent shortcuts conditioned on privileged data, harming cross-domain generalization and exploration. The novel DemoPSD framework proposes selective adoption of teacher guidance at each token by interpolating the teacher and student output distributions via a disagreement-modulated reverse-KL barycenter target. When the teacher-student distribution discrepancy is small, the student follows the teacher closely; when large, the student reverts toward its own prediction to avoid leakage. Theoretical analysis proves DemoPSD reduces privileged information leakage while preserving exploration capacity under dense supervision. Empirically, DemoPSD outperforms prior self-distillation (SDPO) and reinforcement learning (GRPO) baselines on the multi-domain SciKnowEval scientific reasoning benchmark, boosting accuracy by up to 4.2% @16 and increasing training policy entropy by 35-97%. It also generalizes better to out-of-distribution GPQA graduate science questions, where SDPO deteriorates over training. DemoPSD’s adaptive blending of teacher guidance with student autonomy enables improved reasoning, robustness, and mitigates fundamental OPSD failure modes.

Key findings

• DemoPSD improves @16 accuracy on SciKnowEval by 1.7%–4.2% compared with SDPO across biology, chemistry, material science, and physics domains (Table 1).
• DemoPSD maintains 33%–98% higher training policy entropy than SDPO, indicating better preservation of exploration during training (Fig. 1a, Table 3).
• On the out-of-distribution GPQA benchmark, DemoPSD achieves 54.38% average accuracy versus 46.47% for SDPO, showing less performance degradation over training (Table 2, Fig. 3).
• Teacher-student Jensen-Shannon disagreement score is used per token to modulate the blending coefficient αt between teacher and student distributions, controlling leakage attenuation.
• Theoretical guarantee (Theorem 1) that DemoPSD’s weighted reverse-KL barycenter target attenuates privileged information leakage (measured by expected squared log-probability shifts) more than standard OPSD.
• Theorem 2 proves DemoPSD preserves strictly higher entropy than full teacher distillation targets, retaining the student's exploration ability.
• Geometric (reverse-KL barycenter) mixture target distributes probability mass only to tokens supported by both teacher and student, avoiding mode averaging and inflated entropy of arithmetic mixtures.
• A reprompting mechanism selects one correct rollout per prompt group as privileged information for the teacher during training, ensuring reliable teacher contexts.

Threat model

The adversary is the privileged teacher model which has access to privileged information unavailable to the student at test time. It can leak privileged information by influencing token-level teacher distributions, leading the student to learn answer-dependent shortcuts that do not generalize. The student has no privileged context during inference, so replicating these teacher-conditioned distributions fully is impossible and undesirable. Adversaries cannot alter training data or inject malicious prompts outside the teacher-student setup.

Methodology — deep read

The authors formulate the privileged information leakage problem in on-policy self-distillation (OPSD). In OPSD, the teacher is the current model conditioned on privileged information y*, while the student conditions only on the question x.

1. Threat Model & Assumptions: The privileged information y* (e.g., verified reasoning steps or ground truth) is available only to the teacher during training but never to the student at test time. The adversary is the privileged teacher itself that can leak answer-dependent shortcuts to the student if the student is forced to closely mimic the teacher’s conditional distribution. The goal is to reduce this privileged information leakage while benefiting from token-level dense supervision.

2. Data: The training dataset is SciKnowEval, containing multiple-choice scientific questions across four domains (biology, chemistry, material science, physics). The model is trained and evaluated separately per domain. The student model rollouts are filtered to keep only prompt groups with at least one correct rollout; such correct rollouts serve as privileged information y*.

3. Architecture/Algorithm:

• The teacher distribution πT at each token conditions on (x, y*, y<t), while the student distribution πS depends only on (x, y<t).
• Teacher-student disagreement at each token t is measured by the Jensen-Shannon divergence d_t = JSD(πS || πT).
• A leakage attenuation coefficient α_t = f(d_t) is computed using a rescaled sigmoid function that maps disagreement d_t to [0, αmax], controlling how close the distillation target is to teacher or student.
• The reverse-KL barycenter target distribution is defined as: παt_target(v) ∝ πT(v)^(1-α_t) * πS(v)^α_t. This geometric mixture balances mimicking the teacher and retaining the student’s own reasoning.
• The student optimizes the reverse KL divergence to this barycenter target over generated rollouts, with stop-gradient applied to the target distributions and coefficients for stability.
• A reprompting strategy selects one correct rollout as privileged y* for the teacher per prompt group, and training skips groups without correct rollouts.

4. Training Regime: The base model is Qwen3-4B-Instruct. Train on SciKnowEval using batch sampling, generating student rollouts per prompt. For each token, compute disagreement, α_t, and update the student via gradient descent on the reverse KL objective L_DemoPSD(θ). An exponential moving average (EMA) version of the student is used for stable teacher and student distribution estimation in disagreement computation.

5. Evaluation Protocol:

• Metrics include mean@16, majority@16, best@16 accuracy on SciKnowEval validation per domain.
• Training policy entropy is recorded as a proxy for exploration preservation.
• Out-of-distribution generalization is tested on GPQA Extended, a graduate science multiple-choice dataset in biology, chemistry, and physics.
• Baselines are GRPO (group relative policy optimization) and SDPO (standard on-policy self-distillation).
• Ablation studies on sensitivity to hyperparameters β and αmax, and analysis of disagreement scores over domains.

6. Reproducibility:

• Code or checkpoints are not explicitly stated as released.
• The paper provides detailed algorithm pseudocode and hyperparameters.

Concrete end-to-end example: A question x is sampled, multiple rollouts generated from the student model. One correct rollout is sampled as privileged information y*. At each token t in a student rollout, the teacher distribution πT is computed conditioning on (x, y*, student prefix), and the student distribution πS conditioning on (x, student prefix). The JSD disagreement d_t is computed, then α_t via a rescaled sigmoid. The distillation target παt_target is the geometric interpolation between πT and πS controlled by α_t. The student parameters are updated to minimize the reverse KL divergence between π_S and the fixed παt_target across all tokens and rollouts. This process reduces learning from teacher tokens heavily influenced by privileged info while preserving agreement signals, thus avoiding leakage.

Technical innovations

• Introducing a token-level reverse-KL barycenter target that adaptively interpolates between privileged teacher and unprivileged student output distributions using disagreement-based weighting.
• Defining and leveraging teacher-student Jensen-Shannon divergence per token to guide selective attenuation of privileged information leakage in dense token-level self-distillation.
• Proving leakage attenuation (Theorem 1) and exploration preservation (Theorem 2) properties of the disagreement-modulated reverse-KL barycenter objective.
• Using a geometric mixture target instead of arithmetic averaging to avoid mode averaging and inflated entropy, sharpening the distillation signal.
• Employing a reprompting mechanism that selects successful student rollouts as privileged information for teacher conditioning dynamically during training.

Datasets

• SciKnowEval — multi-domain scientific reasoning benchmark with biology, chemistry, material science, physics domains — public
• GPQA Extended — graduate-level science questions in biology, chemistry, physics — public (used for out-of-distribution testing)

Baselines vs proposed

• GRPO: mean@16 accuracy average = 60.49% vs DemoPSD = 65.1% (+4.61%)
• SDPO: mean@16 accuracy average = 63.42% vs DemoPSD = 65.1% (+1.68%)
• SDPO: training entropy lower by 33%–98% compared to DemoPSD across domains
• On GPQA Extended out-of-distribution, SDPO average accuracy = 46.47% vs DemoPSD = 54.38% (+7.91%)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.02502.

Fig 1: DemoPSD preserves higher entropy (left), which translates into better best@16 performance

Fig 2 (page 1).

Fig 3 (page 1).

Fig 4 (page 2).

Fig 5 (page 2).

Fig 2: (a) Validation accuracy curves across four domains of SciKnowEval. (b) Sensitivity to β.

Fig 3: Out-of-distribution generalization on GPQA Extended. Each panel tracks GPQA accuracy over

Fig 4: Disagreement analysis of DemoPSD across four scientific domains.

Limitations

• Experiments are limited to scientific multi-choice QA tasks; generality to other reasoning or generation tasks is untested.
• The analysis assumes positive covariance between privileged signal shifts and student predictions, which may not always hold.
• No adversarial evaluation against malicious privileged information leakage attacks is reported.
• The method relies on availability of reliable privileged correct rollouts for reprompting; situations lacking them will skip distillation steps.
• No ablation on different geometric interpolation functions or alternative disagreement metrics was presented.
• Code release or exact hyperparameters for direct reproduction are not provided, limiting reproducibility.

Open questions / follow-ons

• How does DemoPSD perform on generative tasks or open-ended reasoning benchmarks beyond multi-choice science QA?
• Can alternative disagreement measures or uncertainty metrics improve leakage attenuation further?
• How robust is DemoPSD under distribution shifts where privileged information distributions differ systematically from training?
• What are the trade-offs between exploration preservation and leakage attenuation in larger models or multi-teacher settings?

Why it matters for bot defense

For bot defense and CAPTCHA practitioners, DemoPSD introduces a principled way to mitigate leakage of privileged signals in self-distillation methods that rely on teacher guidance with side information. When leveraging dense token-level supervision or reinforcement learning-based feedback for training language models to analyze or solve CAPTCHA-like puzzles, the risk exists that privileged or oracle information at training induces shortcuts that degrade robustness at test time. DemoPSD’s selective blending of teacher and student distributions at the token level, modulated by their disagreement, can preserve the learner’s independent reasoning and exploration capabilities, crucial for generalized and adaptive bot-detection systems that resist overfitting to specific trapdoor signals. Practical application would involve measuring and controlling privileged information influence embedded in training signals to reduce shortcut exploitation in deployed CAPTCHA solvers or bot detection models.

Cite

@article{arxiv2607_02502,
  title={ DemoPSD: Disagreement-Modulated Policy Self-Distillation },
  author={ Yunhe Li and Hao Shi and Wenhao Liu and Mengzhe Ruan and Hanxu Hou and Zhongxiang Dai and Shuang Qiu and Linqi Song },
  journal={arXiv preprint arXiv:2607.02502},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.02502}
}

Read the full paper

• arXiv:2607.02502 (abstract)
• arXiv:2607.02502 (PDF)