A Penny for Your Prompts: Experiments Detecting and Mitigating LLM Usage by Survey Respondents
Source: arXiv:2607.00403 · Published 2026-07-01 · By Zane Xu, Nathan Malkin
TL;DR
This paper addresses the growing challenge of large language model (LLM) usage by participants completing online surveys on crowdsourcing platforms, which threatens data validity in human-centered security and privacy research. The authors conducted a controlled experimental study with 250 participants across Prolific and Amazon Mechanical Turk (MTurk), testing different conditions such as survey length, explicit instructions against LLM use, and disabling copy-paste functionality to detect and mitigate LLM-assisted responses. They identified behavioral and textual characteristics distinguishing LLM responses, including copy-paste activity, verbosity, and distinct patterns on attention check questions. While mitigation measures reduced LLM use, they did not consistently improve data quality. Additionally, browser-use agents were explored in a separate set of experiments, showing variable but often self-identifying behavior as AI. The study proposes practical detection heuristics and mitigation recommendations grounded in real-world survey settings.
Key findings
- Copy-paste heuristics identified 8% LLM usage on a 5-minute Prolific survey baseline, versus over 80% on MTurk in similar conditions.
- 38.8% of MTurk participants self-reported LLM use; 89.5% of these were also flagged by copy-paste detection.
- LLM-assisted responses were significantly longer and more detailed, with median length for a key open-ended question four words versus two words for humans (U = 4853.5, p < .001).
- Suspected LLM responses disproportionately mentioned scientists for a dinner question (35% vs 1% of human responses), indicating distinctive content patterns.
- Tab switching frequency was higher among suspected LLM users (U = 5740.0, p < .001), but alone had low precision/recall (F1 = 0.545) for detection.
- Mitigation interventions, including explicit requests not to use AI and disabling copy-paste, reduced detected LLM use but in some cases (e.g., disabling copy-paste on MTurk) resulted in lower data quality.
- Human raters’ judgments of LLM-generated text achieved fair agreement (Krippendorff’s α = 0.718) and aligned strongly but imperfectly with detection heuristics.
- Browser-use agents (Claude, Gemini, Perplexity) tested separately often self-identified as AI and refused surveys, limiting adversarial impact during data collection.
Threat model
The adversary is a survey participant aiming to use an LLM or browser-based AI agent to generate or significantly assist survey responses, potentially to maximize compensation with minimal effort. They may use copy-paste, submitting AI-generated text verbatim or near-verbatim. They do not engage in highly sophisticated evasion tactics like retyping responses, nor did browser-use agents widely exist at the time of data collection. The adversary does not have direct control over survey instrumentation or detection algorithms but may attempt to circumvent them by behavioral means.
Methodology — deep read
The study defined LLM usage as any open-ended survey response assisted by an LLM, without distinguishing partial editing versus full generation. Participants (N=250) completed surveys about privacy perceptions of drones on two platforms: Prolific and MTurk. Five conditions were tested: baseline 5-minute on each platform, a 10-minute longer survey on Prolific, a 'no AI' instruction variant on Prolific, and a copy-paste disabled condition on MTurk. The survey included multiple-choice, open-ended, attention check, and comprehension questions. Behavioral metadata was collected via custom JavaScript: keystrokes, tab-switching, mouse activity, timestamps, and copy-paste events, logged alongside survey responses and IP addresses to detect duplicates. Self-reports of LLM use were solicited at the survey end. Qualitative coding was performed on open-ended responses by two researchers, with a codebook constructed and inter-rater reliability statistics computed (e.g., Krippendorff’s α = 0.718 for LLM-usage likelihood ratings). Human raters independently scored responses on detail, polish, and likelihood of LLM use. Statistical tests (Mann-Whitney U, Chi-square, Fisher exact) with Šidák correction for multiple comparisons were applied. Detection heuristics were developed combining copy-paste patterns (requiring copying survey text and pasting external text >10 words), self-reporting, human perception, and behavioral features like tab switching. Browser-use AI agents were separately evaluated for evasive behavior. The study recruited US-based participants with high platform approval rates and compensated all participants fairly. Limitations include lack of ground truth labels for LLM usage, single-topic survey focus, and no testing of all mitigation variants across platforms. Overall, the methodology combined behavioral logging, content analysis, self-report, and human judgment to triangulate LLM usage in realistic survey settings.
Technical innovations
- Developed combined detection heuristics for LLM usage integrating copy-paste behavior and self-reporting rather than relying solely on unreliable AI text classifiers.
- Identified distinctive response patterns on open-ended attention check questions that differentiate LLM-generated text from human answers.
- Demonstrated keystroke and browser event logging as practical behavioral signals for detecting LLM use in online surveys.
- Explored interaction effects between platform choice, survey length, and intervention strategies on LLM usage prevalence and data quality.
Datasets
- Prolific survey responses — 150 participants — collected by authors
- MTurk survey responses — 100 participants — collected by authors
Baselines vs proposed
- Baseline Prolific 5-min survey: ~8% suspected LLM use vs MTurk 5-min baseline: >80% suspected LLM use via copy-paste heuristic
- Self-report LLM use on MTurk: 38.8% vs Prolific: 0%
- Human raters detection agreement (Krippendorff’s α) = 0.718; aligned 80% with core heuristics but with both false positives and negatives
- Tab switching heuristic alone: accuracy 79.9%, F1 score 0.545 (low precision/recall)
Limitations
- No ground truth verification of LLM-generated responses; detection heuristics may yield false positives/negatives
- Single-topic survey on drone privacy perceptions may limit generalizability to other survey types or domains
- Intervention conditions were not fully factorially crossed across both platforms, limiting direct comparison
- Behavioral logging could miss advanced evasion like manual retyping or browser agents that simulate human input
- Data collected before widespread browser-use AI agents; emergent adversarial methods may reduce detection effectiveness
- Restricted to US participants recruited from two major crowdsourcing platforms; other populations might differ
Open questions / follow-ons
- How effective are detection heuristics against advanced browser-use agent automation designed to mimic human typing patterns?
- What are the impacts of LLM usage on survey results across diverse topics and populations beyond privacy/security research?
- Can real-time interventions (e.g., dynamic AI usage warnings, interactive attention checks) further reduce LLM-assisted responding without harming participation or data quality?
- How can more robust ground truth datasets of LLM-assisted survey responses be collected ethically to train automated detectors?
Why it matters for bot defense
Bot-defense and CAPTCHA engineers should note that traditional text-based detectors and attention checks are insufficient to detect or prevent LLM-assisted survey responses because LLMs can easily bypass them. Behavioral signals such as copy-paste activity and tab switching, combined with human-reviewed heuristics, yield more reliable detection. However, detection remains imperfect and evasion tactics may arise. This suggests the need for multi-modal detection approaches that monitor user interactions at the browser level. Also, mitigation approaches—like disabling copy-paste—may degrade data quality and frustrate users, so a balance is needed. Furthermore, the presence of browser-use LLM agents that self-identify and refuse surveys in some cases points to a potential avenue for defensive signaling or verification. Overall, bot-defense in survey contexts requires integrating instrumented behavioral monitoring with tailored instructions and open-ended questions to reduce AI-generated responses while maintaining human usability.
Cite
@article{arxiv2607_00403,
title={ A Penny for Your Prompts: Experiments Detecting and Mitigating LLM Usage by Survey Respondents },
author={ Zane Xu and Nathan Malkin },
journal={arXiv preprint arXiv:2607.00403},
year={ 2026 },
url={https://arxiv.org/abs/2607.00403}
}