Whose Agent Are You? Multi-Layer Fingerprinting and Attribution of Autonomous Web Agents
Source: arXiv:2606.20910 · Published 2026-06-18 · By Dayeon Kang, Hyejun Jeong, Jade Sheffey, Pubali Datta, Amir Houmansadr
TL;DR
This paper addresses the emerging challenge of reliably identifying and attributing AI-powered autonomous web agents that use large language models coupled with browser-level control to browse and interact with websites. Existing defenses like robots.txt and CAPTCHA-based active blocking are inadequate because modern AI agents routinely bypass them. The authors propose MARK, a multi-layer fingerprinting framework that collects features spanning network layers (TLS, HTTP/2) and browser interaction behavior to fingerprint and distinguish AI agents from humans and legacy crawlers. They analyze six prominent web agents including AutoGen, Browser Use, Claude, Gemini, Operator, and Skyvern, uncovering systematic structural differences in their network request patterns, TLS handshakes, HTTP headers, and in-browser behavior. By extracting features across these dimensions and training a decision tree classifier, MARK achieves 97% accuracy in multi-class agent attribution on data collected from a live instrumented website with controlled user interaction tasks. Network-layer features enable early-stage classification, while behavioral signals improve accuracy over time. This work demonstrates a robust, evasion-resistant approach to AI agent detection and attribution using cross-layer signatures rather than relying on brittle heuristics or active blocking mechanisms.
Key findings
- The proposed decision tree classifier achieves 97% accuracy in distinguishing among six AI web agents, humans, and legacy crawlers using multi-layer fingerprint features.
- Temporal request features (Inter-Request Interval, request rate, and Coefficient of Variation) cluster agents into three groups and correspond to their browsing pacing strategies, but are influenced by agent design more than agent type.
- Claude has the lowest temporal request rate variability (CV) among agents, indicating consistent behavior; AutoGen shows high variability.
- Distinctive network-layer TLS fingerprints include unique JA4 extension counts, cipher suites, and HTTP/2 priority weights that differentiate agents such as Skyvern and AutoGen.
- HTTP header-level features reveal bot-indicative signals like Sec-Fetch rule violations in Browser Use and inconsistent User-Agent headers in AutoGen.
- Behavioral features from click, scroll, hover, input, and timing events contribute to separating agents with similar network-layer fingerprints, improving attribution robustness.
- Early agent identification (>60% accuracy) is possible after receiving only three HTTP requests, using mainly network-layer features; behavioral features accumulate over time for higher confidence.
- Four different feature layers (temporal, TLS, HTTP, behavioral) provide complementary information, enabling robust multi-class attribution against evasive AI web agents.
Threat model
Adversaries are autonomous AI web agents controlling real browser instances with large language model backbones, capable of making autonomous browsing and interaction decisions while spoofing common client headers. They cannot fully mask lower-level TLS fingerprints or perfectly replicate human-like browsing behavior, especially under fine-grained multi-layer observation. They are presumed not to have access or capability to alter server-side instrumentation or control the fingerprint extraction pipeline.
Methodology — deep read
The authors first define a threat model where the adversary is an autonomous AI web agent using LLMs to browse websites and perform tasks, capable of simulating browser behaviors but distinguishable due to inherent client and behavioral fingerprints. The framework aims to identify which agent type or class (six AI web agents plus humans and traditional crawlers) is accessing a website under realistic interaction scenarios.
Data collection occurs on a fully instrumented testbed website deployed on AWS EC2 with TLS terminated using Let’s Encrypt certificates. Five controlled UX scenarios probe distinct browser interactions (functional UI testing, state retention, hover reveal, semantic label mismatch, and temporal feedback adaptation). The server logs all incoming TLS handshakes, HTTP/1.1 and HTTP/2 metadata, and per-request timing, while the front-end records detailed user events (mouse move, click, keyboard, scroll, inputs) sampled at millisecond resolution, buffered and sent via beacon to avoid telemetry contamination.
The dataset comprises traces collected from six AI web agents spanning different observation models: open-source DOM- and accessibility-tree-based (AutoGen, Browser Use, Skyvern with GPT-5-mini backbone) and proprietary vision-based or hybrid agents (Claude, Gemini, Operator). Human and legacy crawlers (Nutch, Heritrix, Scrapy) provide baseline comparative data. Each agent performs a scripted multi-page interaction task 30 times to account for stochasticity.
Four independent feature extractors parse timestamps into temporal request statistics (Inter-Request Interval mean, std, coefficient of variation), TLS ClientHello and HTTP/2 connection fingerprints (JA3/JA4 hashes, cipher suites, TLS extension counts, HTTP/2 frame orders and settings), HTTP header-level features (header order hashes, Sec-Fetch-* flag violations, User-Agent coherence), and browser behavioral signals (event rates, inter-event intervals, click and scroll distributions, input latencies, task success signals).
These multi-layer features are aggregated at the trace/session level into a unified feature vector input to a decision tree classifier, designed for interpretability and rapid inference. The classifier is trained on the labeled dataset of agent traces, humans, and crawlers, with hold-out validation on separate runs. Temporal features enable early classification with coarse accuracy, while fine-grained identification improves as behavioral features accumulate.
The paper provides numerous examples such as how Skyvern uniquely differs by TLS extension counts and HTTP/2 priority weights, or how Browser Use frequently violates Sec-Fetch-* request header logic. Claude and Gemini share similar TLS fingerprints but diverge in behavioral patterns due to their hybrid perception-action loop implementations. Operator is harder to distinguish with only network or browsing features but becomes separable after temporal feature analysis.
Code for the logging framework and classifier is not explicitly stated as released, but the authors detail the logging and processing pipelines sufficiently for reproduction, though proprietary agents rely on internal backbones. The dataset is collected on a dedicated domain with strict scenario controls ensuring trace validity. The authors note the importance of multi-layer, multi-modal feature fusion for robust and evasion-resistant attribution.
Overall, this methodology combines passive server-side network telemetry with client-side interaction monitoring in a controlled, repeatable experiment to produce high-fidelity multi-class classification of autonomous AI web agents.
Technical innovations
- A multi-layer fingerprinting framework (MARK) integrating request timing, TLS fingerprinting, HTTP/2 header analysis, and detailed in-browser behavioral events for agent attribution.
- Design of controlled UX scenarios probing subtle interaction differences (e.g., hover-triggered elements, semantic HTML label mismatches, delayed feedback) to surface agent-specific behavioral signatures.
- Application of JA4 TLS handshake fingerprint parsing combined with HTTP/2 priority frame hashing for fine-grained network-layer differentiation of web agents.
- Use of a lightweight decision tree classifier fed with aggregated multi-modal features enabling early-stage and highly accurate (97%) multi-class agent identification.
Datasets
- MARK Dataset — 30 trials each for six AI web agents, human visitors, and legacy crawlers — collected on a dedicated instrumented domain hosted on AWS EC2.
Baselines vs proposed
- Baseline human and traditional crawlers (Nutch, Heritrix, Scrapy): Classified with near 100% accuracy versus AI agents.
- Only temporal features: Initial identification accuracy >60% after three requests.
- Only network-level features (TLS, HTTP): Accurate early-stage differentiation of agents like Skyvern and AutoGen.
- Full multi-layer fusion with decision tree classifier: 97% overall accuracy multi-class attribution.
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.20910.
Fig 1: Overview of MARK. Our Multi-layer Agent fingerprinting framewoRK (MARK) consists of four stages: (1)
Fig 2: Comparison of request timing (Inter-Request-
Fig 3: Comparison results of the request rate sent by
Fig 4 (page 5).
Fig 5 (page 5).
Fig 5: Web component interaction proportion for each web agent on different pages. The event profile proportion
Fig 9: Scenario 1, version 1.
Fig 10: Scenario 1, version 2.
Limitations
- The analysis uses a fixed scripted interaction scenario, which may limit generalizability to arbitrary real-world agent tasks or domains.
- Behavioral features accumulate over several page requests, so immediate classification remains limited (~60%), potentially reducing effectiveness for very short sessions.
- The proprietary agents (Claude, Gemini, Operator) use closed system models, limiting deeper understanding of causative fingerprint sources.
- No explicit adversarial evaluation—authors do not test robustness against agents actively manipulating headers or behavioral signals to evade detection.
- The dataset is limited to six specific agents and three legacy crawlers, so coverage of the full space of web bots is incomplete.
- Fingerprinting accuracy may degrade over time as agent designs evolve, requiring ongoing retraining and scenario updates.
Open questions / follow-ons
- How robust are the multi-layer fingerprints against deliberate adversarial evasion strategies targeting network fingerprint fields or behavioral signals?
- Can the approach generalize to new, unseen AI agents that use different browsing models or architectures beyond the six studied?
- What is the impact of longer-term behavioral tracking across sessions or domains on agent attribution accuracy and privacy?
- Could real-time detection be improved further with more sophisticated machine learning models or additional sensor modalities?
Why it matters for bot defense
For bot-defense and CAPTCHA practitioners, this work presents a compelling server-side fingerprinting approach to reliably and granularly identify autonomous AI web agents without client-side instrumentation or intrusive challenges. By passively collecting network- and behavior-level signals, deployments can move away from brittle binary bot/human heuristics toward more nuanced multi-class attribution, enabling tailored defenses or adaptive content serving. The multi-layer design mitigates simple evasion tactics such as user-agent spoofing or header manipulation, and early-stage detection after few requests can aid in preemptive risk assessment.
However, unlike CAPTCHAs, this approach relies on collecting sufficient interaction traces and request timing data, which may require user engagement and infrastructure to log behavioral events. It is complementary rather than a replacement for active challenges but provides a powerful fingerprint substrate to enrich bot detection pipelines. Practical adoption should consider scenarios diversity and potential fingerprint drift as agent implementations evolve. The detailed TLS and HTTP/2 analysis could integrate with existing WAF or proxy telemetry platforms to improve weaponizing fingerprints for automated bot filtering.
Cite
@article{arxiv2606_20910,
title={ Whose Agent Are You? Multi-Layer Fingerprinting and Attribution of Autonomous Web Agents },
author={ Dayeon Kang and Hyejun Jeong and Jade Sheffey and Pubali Datta and Amir Houmansadr },
journal={arXiv preprint arXiv:2606.20910},
year={ 2026 },
url={https://arxiv.org/abs/2606.20910}
}