Neuron-Aware Data Selection for Annotation-Free LLM Self-Distillation

Source: arXiv:2607.02460 · Published 2026-07-02 · By Zhuowei Chen, Xiang Lorraine Li

TL;DR

This paper addresses the challenge of post-training large language models (LLMs) in specialized domains without access to external supervision or annotated data. Existing annotation-free self-improvement methods either rely on scalar reward signals derived from model outputs or majority voting over multiple rollouts, but suffer from out-of-domain degradation and calibration issues. The authors propose Neuron On-Policy Self-Distillation (Neuron-OPSD), a novel data-centric framework that leverages internal neuron activation patterns to guide both selection of reliable training samples and curation of few-shot context examples to construct teacher distributions for on-policy distillation. By extracting neuron consensus as a proxy for response reliability and neuron overlap to identify contextually relevant demos, Neuron-OPSD trains a student LLM without ground-truth labels, improving domain-specific accuracy while preserving out-of-domain generalization and mitigating calibration collapse. Experiments on multiple scientific and educational benchmarks demonstrate competitive or superior in-domain gains relative to prior annotation-free methods, along with more stable calibration and better accuracy-preservation trade-offs.

Key findings

• Neuron consensus, measured as the number of activated neurons during generation, correlates negatively with hallucination rates: samples with lower activation counts tend to be more reliable (Fig 2)
• Training on both low-activation (bottom 20%) and high-activation (top 20%) subsets yields differing gains; top 20% noisy samples can still provide reasonable improvements in some domains (Table 1)
• Neuron-overlap retrieval of few-shot demonstrations based on Jaccard similarity of activated neuron sets outperforms uniform random retrieval in more diverse domains like Chemistry, particularly improving teacher context relevance (Table 2)
• Neuron-OPSD improves in-domain Avg@8 accuracy by +2.27% on Material and +2.93% on Physics, matching or exceeding TTRL and outperforming LMSI and Intuitor baselines, with stronger calibration stability as measured by lower ECE increments (Tables 3,4)
• Across domains, N-OPSD better preserves cross-domain generalization accuracy and calibration compared to reward-based RL approaches, which inflate calibration error
• Teacher-student token-level sharpness gap (per-token entropy reduction) corresponds to larger in-domain gains, indicating the importance of an informative teacher distribution created by neuron-based context curation (Table 6)
• Neuron consensus alone is insufficient to select the best training data, as both reliability and room for teacher-induced sharpening are needed for useful self-distillation signals
• The EMA (exponential moving average) teacher update stabilizes training by smoothing the teacher parameters derived from the same model

Threat model

The adversary model is 'n/a' as this is not a security paper. However, implicitly the scenario assumes the absence of any external reliable supervision or access to stronger teacher models; the model itself acts as the sole source of supervision, relying only on unlabeled domain data and its internal neuron activation signals to guide self-improvement. The model cannot query humans, obtain real-world interaction feedback, or access externally verified labels. It must avoid reinforcing its own hallucinations and miscalibrations during self-training.

Methodology — deep read

The authors propose Neuron-OPSD, an annotation-free on-policy distillation pipeline harnessing internal neuron activations to guide self-distillation training of LLMs without any ground-truth labels. The threat model assumes no external annotations, no interaction with environments or humans, and no access to stronger teacher or verifier models; the model must self-improve using only unlabeled domain-specific data and its own internal signals. The unlabeled datasets include SciKnowEval (four scientific domains: Biology, Material, Physics, Chemistry; each 80%-20% train-test split), Edu-Feedback (binary classification of feedback quality), and MMLU-Pro (multi-domain multiple choice, 80% data for self-improvement).

The pipeline operates in four stages: (1) Neuron Activation Extraction: For each unlabeled example, the zero-shot rollout is generated by the frozen base LLM, and neuron activations are extracted per generated token by computing early unembedding contribution scores of MLP neurons. The top-K neurons per token are unioned to form a sparse activation set N(x). (2) Sample Selection via Neuron Consensus: The size of N(x) serves as a proxy reliability score s(x). Samples with lower activation counts indicate neuron consensus and are more reliable, while higher counts indicate uncertainty/hallucination.

(3) Neuron Overlap-based Context Curation: For each selected training sample x_q, a few-shot set of m demonstrations is retrieved by minimizing Jaccard distance between activation patterns of x_q and candidate examples, selecting those with most similar neural firing patterns. This creates an augmented input context that elicits a differentiated teacher distribution.

(4) Self-Evolving Training via On-Policy Distillation (OPD): The student is the current base LLM; the teacher is an EMA-smoothed copy of the student conditioned on the neuron-overlap augmented few-shot context. On-policy rollouts from the student produce token sequences; the student is trained to minimize the token-level reverse KL divergence between its next-token distribution and the teacher's context-augmented next-token distribution. The EMA teacher is updated simultaneously to smooth the target distribution.

Training specifics such as epochs, batch size, optimizer, seeds, or hardware are not explicitly detailed. The OPD loss is a sum over token-level reverse KL per token in student-generated trajectories sampled from the student policy. The key innovation is the data-centric use of neuron activation-based heuristics to select training samples and contexts without any external ground-truth. This mitigates the risk of reinforcing incorrect hallucinated behavior.

Evaluation protocol involves sampling 8 rollouts per test query and reporting Avg@8 (mean per-sample accuracy), Maj@8 (majority-vote accuracy), and Expected Calibration Error (ECE) computed from majority vote confidence. Baselines include SFT-based LMSI, on-policy RL TTRL, and Intuitor (annotation-free RL using self-certainty rewards). Cross-domain evaluation measures preservation of generalization to non-source domains. Ablations study the separate effects of neuron consensus filtering and neuron-overlap context.

An example end-to-end: an unlabeled scientific question is passed through the frozen LLM to produce a zero-shot rollout; activated neurons are extracted and counted to estimate reliability; samples with low neuron counts are selected as training data; for each training example, a few most neuron-similar examples are retrieved via Jaccard distance to create a few-shot prompt context; the EMA teacher model conditioned on this context produces a softened token distribution; the student model generates trajectories and trains to reverse-KL match the teacher's token distributions, iterating parameters and updating the EMA teacher periodically.

The method is reproducible in principle given access to the base LLM and unlabeled data, although code or pretrained weights are not explicitly released. Data used is mostly public or standard benchmarks.

Technical innovations

• Leveraging internal neuron activation counts (Neuron Consensus) as a label-free proxy for sample reliability and hallucination detection in LLM self-training
• Using neuron-overlap (Jaccard similarity of neuron activations) to retrieve few-shot demonstrations that induce more informative teacher contexts for on-policy distillation
• Combining these two neuron-activation-based signals to guide both training data selection and teacher context construction in annotation-free on-policy self-distillation
• Applying an EMA teacher updated from the student model with neuron-overlap conditioned context to stabilize token-level distributional distillation without external supervision

Datasets

• SciKnowEval — multiple-choice scientific domain dataset across Biology, Material, Physics, Chemistry — public
• Edu-Feedback — binary classification feedback quality dataset, 1,799 train, 1,000 test samples — public
• MMLU-Pro — multi-domain multiple-choice benchmark — public

Baselines vs proposed

• LMSI: Avg@8 accuracy gain on Biology domain = -2.04% vs Neuron-OPSD = +2.27% on Material domain
• TTRL: Avg@8 accuracy gain on Physics domain = +2.78% vs Neuron-OPSD = +2.93%
• Intuitor: Avg@8 accuracy gain on Biology = +2.29% but worse calibration (higher ECE) vs Neuron-OPSD with more stable or reduced ECE
• Neuron-OPSD cross-domain Avg@8 accuracy gains were +0.61% to +1.31% while LMSI and Intuitor suffered up to -5.11% drops

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.02460.

Fig 1: Overview of the Proposed Neuron-OPSD.

Fig 2 (page 1).

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Limitations

• No explicit reporting of training hyperparameters, compute resources, or random seed strategies limits reproducibility details
• No strong adversarial evaluation or stress tests under distribution shifts beyond cross-domain generalization reported
• Calibration benefits are not uniform; N-OPSD can increase Expected Calibration Error (ECE) on some SciKnowEval domains
• Neuron consensus alone is insufficient as a data selection criterion; the combined effect depends on dataset properties and teacher-student distribution gaps
• The method requires access to neuron activations internally, limiting applicability to open-source or introspectable LLMs
• Experiments focus on medium-scale Qwen-4B model; results might differ for much larger or smaller LLMs

Open questions / follow-ons

• How well do neuron-consensus and neuron-overlap signals generalize to much larger LLMs or different model architectures?
• Can these neuron-activation based selection methods be combined with limited human annotations for semi-supervised learning?
• How does the method perform when the unlabeled pool contains significantly out-of-domain or adversarial samples?
• Would integrating uncertainty estimation techniques alongside neuron features further improve sample selection robustness?

Why it matters for bot defense

For practitioners in bot-defense and CAPTCHA-like challenges, Neuron-OPSD introduces an innovative unsupervised self-distillation strategy for LLMs operating in constrained or specialized domains without annotation costs. Its data selection mechanism based on internal neuron activations provides a novel proxy for model confidence and hallucination detection, which could inspire more principled ways to identify reliable or uncertain queries during interaction with bots or adversarial users. The use of neuron overlap to curate context for conditioning teacher distributions also suggests new label-free methods to elicit informative signals from models when external supervision or feedback is unavailable or expensive. While the paper targets domain adaptation and self-improvement, the core idea of leveraging internal neuron statistics for confidence-aware self-supervision aligns with challenges in CAPTCHA anti-abuse systems, where relying solely on external ground truth is infeasible, and internal model signals can provide fresh avenues for improving reliability and robustness against hallucinating or adversarial model outputs.

Cite

@article{arxiv2607_02460,
  title={ Neuron-Aware Data Selection for Annotation-Free LLM Self-Distillation },
  author={ Zhuowei Chen and Xiang Lorraine Li },
  journal={arXiv preprint arXiv:2607.02460},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.02460}
}

Read the full paper

• arXiv:2607.02460 (abstract)
• arXiv:2607.02460 (PDF)