Detecting LLM-Generated Tokens in Human--LLM Coauthored Text

Source: arXiv:2607.21458 · Published 2026-07-23 · By Yangjun Lu, Hongyi Zhou, Fabian Spill, Kai Ye, Chengchun Shi, Jin Zhu

TL;DR

This paper addresses the fine-grained challenge of detecting exactly which tokens in a mixed human-LLM coauthored text are generated by an LLM, rather than simply classifying the entire document as human- or machine-written. Existing detection methods mostly operate at the document level or predefined units like sentences, limiting their utility for localizing model contributions when human and AI writing is interleaved at sub-sentence granularity. The authors propose a novel token-level detection approach: first computing existing token-level detection scores, then smoothing these scores locally across neighboring tokens with kernel smoothing. Crucially, they use an adaptive bandwidth selection method based on Lepski’s rule to balance smoothing bias and variance depending on local authorship structure. This method requires no token-labeled training data and can flexibly work directly at the token level. Theoretically, they derive bounds on the mean squared error of the smoothed score estimates and prove oracle-like guarantees for the adaptive bandwidth selection. Empirically, the method consistently outperforms multiple baselines on both synthetic and real human-LLM coauthored datasets across several language models and collaboration patterns, achieving up to 94% accuracy in token authorship prediction in a realistic case study. A publicly available online demo implements the approach to analyze real texts.

Key findings

• Kernel smoothing of token-level detection scores improves median AUC over raw scores across Writing, XSUM, and ESSAY datasets by up to 30 points (e.g., raw 0.60 vs auto kernel ~0.94 in some cases).
• The triangular kernel outperforms the uniform kernel in reducing contamination bias near authorship boundaries, improving token-level detection accuracy by ~2-5% AUC.
• Adaptive bandwidth selection via Lepski’s rule generally improves performance over all fixed bandwidth selections and approaches oracle bandwidth performance.
• On realistic human–LLM coauthored abstracts, the method achieves 94% token-level authorship classification accuracy.
• In comparisons across three black-box LLMs (Gemini-3.1, Grok-4.1, Claude-4.5) and three datasets, the proposed method yields median document-level AUC of 0.842 to 0.887, outperforming baselines such as DeBERTa, SegFormer, SenPred, TextTiling, and TriBERT.
• Theoretical analysis establishes upper bounds on mean squared error of kernel-weighted token score estimates capturing bias from smoothing across authorship boundaries and variance from sample size and correlations.
• The Lepski-style bandwidth selector provably adapts to local authorship boundary structure to achieve near-optimal bias-variance trade-off without token-level supervision.
• The method is compatible with diverse underlying token-level detection scores (e.g. Fast-DetectGPT, AdaDetectGPT) and requires no new training data.

Threat model

The adversary is a human-LLM coauthoring scenario where the text contains interleaved human- and LLM-generated tokens with unknown authorship labels. The defender does not assume access to token-level ground truth or any LLM internal states beyond token-level scores computed by pre-existing detectors. The adversary cannot alter detection scores or labels but might compose texts with varying segment lengths and intra-sentence interleaving. The threat is detecting AI contributions fine-grainedly; not considered is an adaptive adversary crafting adversarial token sequences to evade detection.

Methodology — deep read

The threat model assumes documents are coauthored by humans and an LLM, generating interleaved segments of text with unknown token-level authorship labels (human or LLM). The adversary is not explicitly modeled; the goal is to estimate token authorship from the observed text alone. They do not assume access to token-level ground truth labels for training. The data consists of four datasets: three synthetic human-LLM coauthored corpora constructed by concatenating human-written sentences with LLM-generated continuations (Writing, XSUM, ESSAY; 100 documents each), plus a realistic coauthored abstract dataset. Token-level ground truth authorship is derived from the synthetic generation process. Token-level detection scores (St) are computed using pre-existing detection functions Φt such as log probabilities or learned token scores (e.g. AdaDetectGPT). These scores differ in expectation between human and LLM tokens (Assumption 3.1). To reduce noise, token scores are smoothed locally using kernel-weighted averages over adjacent tokens, with weights defined by a non-increasing kernel function K(·), typically triangular due to robustness to boundary contamination. The bandwidth k controls the smoothing window size. They design an adaptive bandwidth selector based on Lepski’s method: multiple candidate bandwidths {k1, ..., kM} are evaluated by computing confidence intervals for smoothed scores, and the largest bandwidth consistent with all smaller ones is selected for each token, allowing local adaptation to authorship boundary structure. Thus, smoothed scores bSt,k are used as token-level authorship indicators, with a threshold η chosen to minimize within-cluster variance of the smoothed scores. The final classifier labels tokens as LLM-generated if bSt,k ≥ η. Theoretically, they prove an upper bound on the mean squared error of bSt,k as an estimator of the latent authorship signal µCt (Thm 4.1), decomposed into bias (from smoothing across boundaries) and variance (inversely proportional to effective sample size, adjusted for token-score correlations). Corollary 4.2 obtains explicit bias-variance formulas for uniform and triangular kernels. Theorem 4.3 proves that the adaptive Lepski bandwidth selector performs near-optimally compared to the best fixed bandwidth. Empirical evaluation uses the synthetic datasets generated by partitioning texts into alternating human-written and LLM-generated sentence blocks with random segment lengths q. Token-level AUC is the primary metric. Ablation studies compare raw score, fixed uniform and triangular kernels, adaptive bandwidth selection, and an oracle bandwidth selector. Baselines include DeBERTa, SegFormer, SenPred, TextTiling, and TriBERT adapted for token-level localization. Experiments run on multiple LLMs including google/gemma-2b-it (open source) and closed models Gemini-3.1, Grok-4.1, and Claude-4.5. A case study applies the method to a real abstract with human-LLM coauthoring, achieving 94% token-level accuracy. The method requires no token-labeled training data and publicly releases code and a demo website for reproducibility.

Technical innovations

• Introducing kernel smoothing of token-level detection scores with adaptive bandwidths for fine-grained localization without token-level supervision.
• Applying Lepski’s method for adaptive bandwidth selection to balance bias and variance locally in authorship detection.
• Deriving theoretical MSE bounds decomposing bias due to smoothing across authorship boundaries and variance from sample size and token-score correlations.
• Developing a thresholding scheme that minimizes within-cluster variance of smoothed token scores to robustly classify token authorship.

Datasets

• Writing — 100 documents — synthetic human-LLM coauthored constructed from Fan et al. 2018 human texts
• XSUM — 100 documents — synthetic human-LLM coauthored constructed from Narayan et al. 2018 news dataset
• ESSAY — 100 documents — synthetic human-LLM coauthored constructed from Verma et al. 2024 educational texts
• Realistic human-LLM coauthored abstracts — size unspecified — publicly available dataset based on He et al. 2016

Baselines vs proposed

• DeBERTa: median document AUC ranges 0.741–0.833 vs proposed: 0.842–0.887
• SegFormer: median document AUC ranges 0.616–0.893 vs proposed: 0.842–0.887
• SenPred: median document AUC ranges 0.528–0.765 vs proposed: 0.842–0.887
• TextTiling: median document AUC ranges 0.522–0.559 vs proposed: 0.842–0.887
• TriBERT: median document AUC ranges 0.706–0.776 vs proposed: 0.842–0.887
• Ablation: raw score baseline median AUC ~0.60 vs fixed triangular kernel smoothing ~0.85 vs adaptive kernel smoothing up to 0.95 in synthetic datasets

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.21458.

Fig 4: A snapshot of the demonstration website.

Fig 5: The snapshot of the detection results for the human–LLM coauthored abstract based on He et al. [2016].

Fig 7: The snapshot of the detection results for the original abstract from He et al. [2016].

Limitations

• No explicit adversarial robustness evaluation against adaptive attackers or sophisticated paraphrasing.
• Synthetic datasets simulate authorship boundaries at sentence-level; performance may degrade if authorship changes occur at finer granularity.
• Assumes token-level detection scores are separable in expectation between human and LLM tokens, which may not hold uniformly across domains or models.
• The method requires tuning candidate bandwidth grids and kernel parameters that may be sensitive to dataset characteristics.
• Closed-source LLMs used pose reproducibility challenges despite open code release for detection method.
• Local smoothing may bias detection near very short or frequent authorship switches below minimum segment length τ.

Open questions / follow-ons

• How robust is the approach to adversarially modified LLM outputs or paraphrasing attacks designed to camouflage AI generation?
• Can this approach be extended or combined with supervised token-level classifiers to leverage limited labeled data?
• How would the method perform under more complex multi-author settings beyond human vs single LLM distinctions?
• What is the impact of tokenization schemes and differing LLM vocabulary/model families on detection score separability?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this paper provides an effective unsupervised method to localize LLM-generated text at the token-level within collaboratively authored documents, which can assist in assessing AI involvement granularity beyond document-level flags. The adaptive kernel smoothing framework offers a principled way to reduce noise in token-level detection scores and adapt to authorship boundary dynamics without requiring costly token-labeled datasets. Such fine-grained detection could enable more nuanced analyses of AI content usage in sensitive contexts like academic integrity, content moderation, or legal compliance. However, practical adoption needs attention to computational costs of token-level smoothing over long texts and consideration of robustness under adversarial conditions typical in bot-generation scenarios.

Cite

@article{arxiv2607_21458,
  title={ Detecting LLM-Generated Tokens in Human--LLM Coauthored Text },
  author={ Yangjun Lu and Hongyi Zhou and Fabian Spill and Kai Ye and Chengchun Shi and Jin Zhu },
  journal={arXiv preprint arXiv:2607.21458},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.21458}
}

Read the full paper

• arXiv:2607.21458 (abstract)
• arXiv:2607.21458 (PDF)