When Agents Learn to Be You: Benchmarking Privacy Leakage, Impersonation Risk, and Defenses in Persona Skills
Source: arXiv:2608.03700 · Published 2026-08-04 · By Yongli Xiang, Zhifang Zhang, Bojun Yang, Ziming Hong, Lei Feng, Miao Xu et al.
TL;DR
This paper addresses the emerging privacy and impersonation risks posed by persona skills, which distill personal interaction histories into portable, executable artifacts that downstream agents use for personalized responses. Unlike prior personalization methods that rely on direct context retrieval or model parameter modifications, persona skills concentrate fragmented personal signals into reusable modules, amplifying privacy leakage and behavioral impersonation threats. To systematically study these risks, the authors introduce AntiSkillBench, a comprehensive benchmark including a dataset of 7,500 persona-grounded dialogue traces from 50 richly profiled users, an evaluation suite measuring privacy leakage and impersonation at the skill and agent level, and a defense suite testing active and passive mitigation strategies.
Key findings
- Skill-level privacy leakage persists across three distillation protocols (Direct, Three-stage, Colleague), with GPT 5.4 reaching overall skill coverage scores between 55.2 and 66.2, showing heavy retention of communication (up to 92%) and personality traits (up to 75.7%).
- Agent-level impersonation risks manifest strongly: GPT 5.4 equipped with Three-stage Distill skill achieves 56.0% Field QA accuracy across demographics, background, personality, and communication and VocabGain up to 31.3, indicating close imitation of target user linguistic style.
- Persona skills leak not only explicit demographic attributes but also behavioral traits such as communication style and personality, which translate into authentic agent-like behavior replicating the user.
- Existing defenses have limited effectiveness and depend heavily on the distillation method: active defenses (Privacy Sanitization and Adversarial Obfuscation) reduce surface-level leakage but fail to suppress deeper trait disclosure, while passive Semantic Backdoor Injection works under direct distillation but degrades under abstraction-based methods.
- Skills distilled from larger numbers of dialogues increase privacy and impersonation risks, indicating a tradeoff between personalization depth and security.
- Impersonation vulnerability is consistent across three leading agent backbones: GPT 5.4, Gemini 3.6 Flash, and Claude Haiku 4.5.
- Semantic backdoor defenses can detect unauthorized skill reuse but do not prevent initial leakage or behavioral mimicry.
- The AntiSkillBench dataset and evaluation suite provide a standardized, multi-dimensional framework for assessing persona skill safety across the pipeline from trace collection to downstream agent interaction.
Threat model
Adversaries can either directly inspect the distilled persona skill artifact to extract private user attributes (skill-level leakage) or interact with an agent equipped with the skill by probing via queries and analyzing responses to infer persona information (agent-level impersonation). The adversary aims to recover explicit identifiers, background, latent personality traits, and communication styles. The adversary does not have capability to modify the skill distillation function or the deployed agent code but can submit queries and observe responses or analyze exported skill artifacts.
Methodology — deep read
The study begins by defining the threat model where an adversary can access either the distilled persona skill artifact directly (skill-level leakage) or interacts with an agent equipped with the skill (agent-level impersonation). The adversary seeks to infer explicit and latent personal attributes including demographics, background, personality, and communication style.
The dataset comprises 50 user profiles sampled from OpenCharacter, enriched with demographic, background, Big Five personality traits, and fine-grained communication style information. For each profile, 50 persona-grounded queries are generated via an LLM conditioned on user attributes, spanning general assistance, tool design, and mathematical calculation domains. Each query is expanded into a three-turn dialogue with a simulated assistant, yielding 2,500 dialogue traces totaling 7,500 user turns.
Three skill distillation protocols process these dialogue traces into persona skills. Direct Distill synthesizes an executable skill from full user dialogue history focusing on explicit language cues. Three-stage Distill extracts broad user attributes, induces conditional behavioral rules, and composes the skill from these parts. Colleague Distill, adapted from COLLEAGUE.SKILL, builds layered skills emphasizing stable character traits, reasoning patterns, and in-character style.
The defense suite includes Privacy Sanitization (PS), an active online method that rewrites user queries into privacy-neutral expressions; Adversarial Obfuscation (ADV), a post-hoc active method that appends follow-up turns injecting conflicting private attributes to mislead distillation; and Semantic Backdoor Injection (SBD), a passive defense embedding semantically triggered watermarks into traces to enable tracing unauthorized skill reuse.
Evaluation metrics are (i) Skill Coverage (SC), measuring how many persona attributes from each user profile are retained in the distilled skill artifact via LLM judgment; (ii) Field QA Accuracy, assessing an agent’s ability to explicitly reveal target user attributes in direct queries; and (iii) VocabGain, quantifying how closely the agent replicates the user’s lexical and discourse style across persona-relevant scenarios, normalized against no-persona and oracle-profile baselines.
Experiments run three agent backbones (GPT 5.4, Gemini 3.6 Flash, Claude Haiku 4.5) both for skill distillation and downstream response generation using identical backbones per experiment. GPT 5.4 also functions as the LLM judge. The 50 user roles with 50 dialogues each compose the full evaluation dataset. Ablations vary dialogue quantity to assess impact on leakage and defense.
Results consistently demonstrate substantial skill-level information retention and agent-level impersonation capability across models and distillation protocols. Defense effectiveness is partial and distillation-dependent: active defenses reduce some leakage but fail to protect deep latent traits, and backdoor defenses enable tracking but not leakage prevention. Quantitative results including skill coverage >55%, QA accuracy >50%, and positive VocabGain confirm persistent privacy and behavioral risks. The code release and detailed dataset description are provided for reproducibility, though some components rely on LLM labeling not fully deterministic.
An example end-to-end flow: A user profile with demographic and personality is used to generate 50 personalized questions. Multi-turn dialogues simulate realistic interaction, forming Tu. Using Three-stage Distill, the skill artifact su is constructed by extracting attributes, inducing behaviors, and creating a reusable persona skill. The skill-equipped GPT 5.4 agent responds to queries x, producing outputs y reflecting user style and preferences. Adversarial attacks may interact with y or directly inspect su to extract personal info. Defenses like PS rewrite queries q in real-time to reduce leakage before distillation. Metrics quantify recovery of attributes from su and imitation in y, highlighting impersonation risk.
Technical innovations
- Introduction of AntiSkillBench, the first end-to-end benchmark targeting privacy and impersonation risks in persona skill distillation pipelines.
- Formulation of dual-level risk assessment combining skill-level privacy leakage and agent-level behavioral impersonation metrics tailored to persona-grounded skills.
- Development of a rich persona dataset incorporating Big Five personality traits and communication style beyond standard demographics for more nuanced risk evaluation.
- Design of novel defense strategies spanning active trace transformations (Privacy Sanitization, Adversarial Obfuscation) and passive semantic backdoors for traceability of skill misuse.
Datasets
- AntiSkillBench — 7,500 dialogue traces originating from 50 user profiles with multi-turn dialogues — synthetic dataset constructed for this study based on OpenCharacter profiles
Baselines vs proposed
- GPT 5.4 Direct Distill: Skill Coverage = 63.6 overall vs Three-stage Distill: 66.2; QA Accuracy Direct Distill = 54.2% vs Three-stage Distill = 56.0%
- Gemini 3.6 Flash Colleague Distill: Skill Coverage = 61.2 vs Three-stage Distill: 65.3; VocabGain Colleague Distill = 40.0 vs Three-stage Distill = 37.6
- Claude Haiku 4.5 Direct Distill: QA Accuracy = 49.8% vs Three-stage Distill: 52.5%; VocabGain Direct Distill = 19.5 vs Three-stage Distill = 19.9
- Defense Privacy Sanitization (online) reduces GPT 5.4 Direct Distill Skill Coverage from ~63.6 to 51.7 and QA Accuracy from ~54.2 to 40.6, but leaves residual leakage.
- Semantic Backdoor Injection effective under Direct Distill but degrades under Colleague Distill, showing defense dependence on distillation strategy.
Limitations
- Synthetic dialogue dataset may not capture full complexity or variability of real-world persona interactions.
- Evaluation uses LLM-based judges for metrics which may introduce subjectivity or lack full reproducibility.
- No comprehensive adversarial robustness evaluation against proactive attackers attempting to remove or bypass defenses.
- The study focuses on three distillation protocols and three agent backbones; results may vary with other architectures or newer models.
- Some defense techniques rely on trace interventions only, without modifying distillation or agent behavior, limiting defense scope.
- Backdoor detection requires access to outputs triggered by semantic patterns, which may be hard to enforce in open settings.
Open questions / follow-ons
- How can defenses be designed that simultaneously reduce latent personality and communication style leakage without sacrificing skill utility?
- What are effective methods to detect and mitigate adversarially crafted skill distillation attacks or poisoning?
- Can the integration of defense mechanisms into the distillation function itself improve generalization across distillation protocols?
- How do persona skill risks translate to downstream tasks in multi-agent or cross-domain contexts with more complex skill compositions?
Why it matters for bot defense
For bot-defense and CAPTCHA practitioners, this work highlights that persona skills—the reusable, distilled modules encoding user attributes and behaviors—can be a potent vector for privacy leakage and impersonation beyond traditional data retrieval or model fine-tuning. Defenses focused only on surface-level data anonymization may be insufficient against the amplified and persistent risks introduced by skill distillation and reuse. Evaluations like those in AntiSkillBench provide rigorous tools to benchmark the privacy and authenticity vulnerabilities at multiple pipeline stages, informing stronger, multi-layered protections. Practitioners designing CAPTCHAs or bot-detection systems that rely on behavioral patterns or personalized responses should be aware that persona skills could potentially spoof user identity or leak sensitive traits, necessitating detection mechanisms tuned to these novel artifacts and the behavioral cues they produce.
Cite
@article{arxiv2608_03700,
title={ When Agents Learn to Be You: Benchmarking Privacy Leakage, Impersonation Risk, and Defenses in Persona Skills },
author={ Yongli Xiang and Zhifang Zhang and Bojun Yang and Ziming Hong and Lei Feng and Miao Xu and Tongliang Liu },
journal={arXiv preprint arXiv:2608.03700},
year={ 2026 },
url={https://arxiv.org/abs/2608.03700}
}