What Does It Take to Detect an AI Agent? Minimal Feature Sets for Behavioral Detection under Browser Automation

Source: arXiv:2607.26935 · Published 2026-07-29 · By Vishisht Choudhary, Lukas Schmidt, Anne Zoë Kenntner, Feras Skhab, Michel Osswald, Jens Ernstberger

TL;DR

This paper tackles the emerging challenge of detecting AI-driven web browsing agents that operate via browser automation frameworks like Playwright. Traditional bot detection assumes a binary classification of traffic as human or bot, but AI agents occupy an intermediate class that resembles neither cleanly. The authors demonstrate that binary classifiers fundamentally fail to represent AI agents, with models like MLP and SAINT misclassifying about 34–39% of agent sessions as human due to the lack of an explicit agent class. Introducing a dedicated third "agent" class in the classifier completely closes this detection gap, yielding perfect per-class F1 scores for agents on a controlled benchmark.

The key discriminative signal arises not from the reasoning or cognition of the agents but from artifacts embedded in the browser automation APIs. Playwright-generated event streams lack raw pointer-event data characteristic of physical device input, leading to telltale signatures such as mouse events without preceding movement and absent wheel events. These signatures persist even under strong evasion attempts including GAN-generated trajectories and replay of real human cursor data. Through exhaustive feature subset search over 17 behavioral features, the authors find minimal two-feature sets sufficing for flawless agent recall and near-perfect precision, enabling practical, robust detection with minimal instrumentation.

Key findings

• Binary classifiers misclassify AI agents as human at high rates: MLP 39.1%, SAINT 34.5%, random forest 30.0% (Table 2).
• Adding an explicit agent class in a three-class framework yields perfect agent F1=1.000 across all 30 runs (3 models × 10 seeds) (Section 6).
• A five-level evasion ladder including GAN-generated and human replay trajectories (n=2299 test sessions) yields zero agent misses in 22,990 per-seed predictions by three model families (Section 7, Table 3).
• Two behavioral features (mouse_event_rate and teleport_click_ratio) alone achieve 100% observed agent recall at every evasion level with agent precision 0.994 (Section 9).
• Five features lift macro-F1 to 0.991 for full three-class discrimination, demonstrating redundant encoding of the signal (Section 9).
• Agent-specific "pause-think-act" features hypothesized to capture LLM latency did not survive backward elimination; detection relies on event stream structure (Section 4.1).
• Task-domain ablation confirms agent detection is invariant to task-specific confounds; agent F1=1.000 across feature tiers with or without task-correlated features (Section 8).
• Detection artifacts stem from Playwright emitting programmatic DOM events lacking raw device signals, causing absence of wheel deltas and pre-click mousemove streams (Section 7).

Threat model

The adversary is an AI web browsing agent controller who seeks to evade detection as human while performing real browsing tasks through Playwright browser automation. They have full knowledge of the features but only black-box access to the detector model (no weights or gradients). Their evasion strategies are constrained to Playwright API-level modifications and do not extend to OS-level event injection or hardware input simulation, which are considered out of scope.

Methodology — deep read

1. Threat model and assumptions: The attacker is a white-box aware adversary who knows all feature definitions but has only black-box access to the detection model and no access to model weights or gradients. They control an AI agent browsing via Playwright automation API and cannot inject events at OS/device level or spoof hardware inputs. Evasion is constrained to Playwright-level manipulations.

2. Data: The dataset combines three sources: over 14,000 real human CAPTCHA-solving sessions from CaptchaSolve30k, 5,000 synthetic bot samples across three archetypes (headless, scripted, sophisticated) sampled from parameterized distributions to mimic real bot variability, and 1,025 real AI agent sessions collected from Claude agent performing 10 concrete web tasks driving Chrome via Playwright. The merged dataset has 6,025 samples (3,000 human, 2,000 bot, 1,025 agent) with 29 features (17 continuous behavioral, 7 binary environment, 5 metadata). Stratified downsampling produces a balanced set of 3,075 samples (1,025 per class) for training/validation/testing splits with seed=42.

3. Feature engineering: They extract 17 continuous behavioral features spanning trajectory entropy, movement efficiency, teleportation, click statistics, timing metrics, and agent-specific ratios like teleport_click_ratio. Seven binary environment features capture browser environment signals such as absence of mouse or wheel events and webdriver flags (excluded from subset analysis due to easy spoofing). Features are min-max normalized based on training split.

4. Model architectures: Multiple model families are trained, including MLP, Random Forest, XGBoost, and SAINT—a tabular transformer adapted for three-class classification with per-feature embeddings and attention blocks. SAINT is pretrained with a denoising objective. Gradient Boosting Machines (GBMs) are used for efficient exhaustive search due to computational cost of transformer training.

5. Training regime: Models are trained with fixed train/val/test splits (70/15/15), over 10 random seeds (42-51) to ensure robustness. GBMs use 100 estimators with max depth 4. The SAINT model uses two transformer blocks of 4 attention heads, linear projections of features with CLS token, pretrained with 30% feature masking prior to fine-tuning.

6. Evaluation: Metrics include per-class precision, recall, and macro F1 score. Binary human-vs-bot classifiers are evaluated on human+bot train data and tested on held-out test and agent data to assess agent miss rate. Three-class classifiers distinguish human, bot, and agent directly. A 5-level evasion ladder tests robustness, escalating from native agent behavior through rule-based cursor smoothing, GAN-generated mouse trajectories, GAN+ with micro-jitter augmentation, and replay of real human cursor data warped to tasks. Over 9,400 GBMs are trained and evaluated over feature subsets of size 1 through 5 to find minimal subsets that preserve agent detection. Statistical variability is reported over multiple seeds.

7. Reproducibility: The paper describes detailed dataset construction, feature extraction, models, and training protocols to enable replication. Some external datasets like CaptchaSolve30k are publicly available; agent sessions and bot generators are described in detail. Implementation details for the evasion ladder, feature computations, and model hyperparameters are provided in appendices. The paper does not explicitly mention public code or frozen weights for the complete end-to-end pipeline.

End-to-end example: To detect agents, the authors first show that a binary SAINT classifier trained on human and bot data misclassifies ~34.5% of agents as human because agents fall outside the learned binary label space. By retraining SAINT with an explicit agent class using the unified dataset (human, bot, agent), the model achieves perfect agent recall (F1=1.000). They then challenge this detector across evasion versions, including realistic GAN-generated and replayed mouse trajectories. Detection remains perfect because key features like mouse_event_rate and teleport_click_ratio capture automation API artifacts (like absence of raw pointer-event teleports before clicks) that cannot be eliminated by evasion within the Playwright threat model. Exhaustive subset search confirms that only two behavioral features are needed to maintain perfect recall and 0.994 precision, enabling minimal, deployable agent detection models.

Technical innovations

• Identification and formalization of a three-class detection framework (human, bot, agent) that eliminates the structural detection gap inherent in binary classification of AI agents as humans or bots.
• Construction of a five-level composable evasion ladder for AI-driven browser automation agents spanning native behavior, rule-based smoothing, GAN-generated trajectories, to replay of real human cursor data, enabling rigorous robustness evaluation.
• Detection signal attribution to browser automation API artifacts (e.g., absent raw pointer-move and wheel-delta event streams, lack of pre-click mousemove teleports) rather than agent reasoning signatures.
• Exhaustive feature subset search (9,401 Gradient Boosting Models over subsets of size 1-5) revealing minimal 2-feature behavioral signatures (mouse_event_rate and teleport_click_ratio or alternatively mouse_event_rate and click_duration_std) sufficing for perfect agent recall under all evasion levels.
• Adaptation and pretraining of the SAINT tabular transformer architecture for multi-class behavioral traffic classification with per-feature embeddings and denoising masking.

Datasets

• CaptchaSolve30k — 14,000+ sessions — public human CAPTCHA-solving mouse dynamics dataset
• Synthetic bots — 5,000 samples balanced across headless, scripted, and sophisticated archetypes — generated via parameterized Gaussian distributions
• Claude agent sessions — 1,025 sessions — real AI agent driving Chrome via Playwright across 10 web tasks

Baselines vs proposed

• MLP (binary human-vs-bot): agent miss rate = 39.1% vs 3-class SAINT: agent F1 = 1.000
• SAINT (binary): agent miss rate = 34.5% vs 3-class SAINT: agent F1 = 1.000
• Random Forest (binary): agent miss rate = 30.0% vs 3-class GBM subset model (2 features): agent recall = 100%, agent precision = 0.994
• XGBoost (binary): agent detection = 100% but via coincidental brittle features (absent scroll/wheel events)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.26935.

Fig 2: Exhaustive search: total subsets tested vs. subsets

Fig 3: Forward selection trajectory. The greedy search

Fig 4: visualizes the full F1 trajectory as features are removed.

Fig 5: Feature importance ranking. The top features com-

Fig 6: Feature distributions across the three traffic

Limitations

• The threat model only covers evasion within Playwright-driven browser automation; OS-level input injection or hardware-based spoofing is outside scope and may circumvent detection.
• Agent behavioral data is collected only from a single AI agent (Claude) driving Chrome, which may limit generalizability to other agents or browsers.
• Synthetic bot data is generated from parameterized distributions, not real-world bot traffic, which may not capture all bot behavior nuances.
• Binary environment features are excluded from minimal feature analysis as they can be trivially spoofed by setting browser properties.
• The dataset and detection rely on behavioral features aggregated at session-level; finer-grained temporal dynamics or newer interaction paradigms are unexplored.
• No analysis of distributional shift over time or different web tasks beyond the ten considered; generalization under evolving browser automation APIs is untested.

Open questions / follow-ons

• How effective are these detection methods against OS-level input simulation or hardware-emulated mouse events that can produce raw pointer streams indistinguishable from human inputs?
• How do different AI agents, browsers, or automation APIs (e.g., Puppeteer, Selenium) affect the universality of the detected artifacts?
• Can detection models be adapted to streaming or continuous monitoring scenarios beyond aggregated session-level features?
• What are the privacy implications and mitigation options when deploying such detailed behavioral feature extraction at scale?

Why it matters for bot defense

For bot-defense engineers and CAPTCHA practitioners, this work highlights a critical structural flaw in existing binary bot detection systems regarding emerging AI web agents. It establishes the necessity of incorporating an explicit "agent" classification to avoid misrouting or misclassifying AI-driven automated traffic. The identification of minimal two-feature behavioral signatures that capture automation API artifacts allows lightweight, robust agent detection without large feature overhead, facilitating scalable deployment.

Additionally, the comprehensive evasion ladder evaluation demonstrates strong resistance to sophisticated cursor-biometrics obfuscation, signaling that browser-level event stream analysis remains a practical leverage point. However, teams must consider that this detection hinges on browser automation API artifacts rather than inherent AI reasoning patterns, so defenses may need adaptation as automation frameworks evolve or new input spoofing methods appear.

Cite

@article{arxiv2607_26935,
  title={ What Does It Take to Detect an AI Agent? Minimal Feature Sets for Behavioral Detection under Browser Automation },
  author={ Vishisht Choudhary and Lukas Schmidt and Anne Zoë Kenntner and Feras Skhab and Michel Osswald and Jens Ernstberger },
  journal={arXiv preprint arXiv:2607.26935},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.26935}
}

Read the full paper

• arXiv:2607.26935 (abstract)
• arXiv:2607.26935 (PDF)