Broken Gates: Re-evaluating Web Bot Defenses in the Age of LLM Agents

Source: arXiv:2607.18659 · Published 2026-07-21 · By Behzad Ousat, Nikita Turkmen, Lalchandra Rampersaud, Dillan Bailey, Amin Kharraz

TL;DR

This paper systematically evaluates the resilience of contemporary web bot defenses against two emerging attacker classes: commercial Captcha-solving services and large language model (LLM)-based browser agents. Traditional anti-bot systems rely on challenge-based (e.g., hCaptcha, reCaptcha v2) and non-interactive trust-based (e.g., reCaptcha v3, Cloudflare Turnstile) mechanisms to distinguish humans from bots. The study comprehensively tests seven commercial solver services and six LLM-based agents across multiple real-world deployed bot defenses. The key insight is that while commercial solver services achieve near-perfect success defeating challenge-based Captchas at very low cost, their effectiveness degrades substantially against non-interactive behavioral defenses like reCaptcha v3. LLM-based browser agents largely fail to bypass defenses unless equipped with dedicated solver modules or operating within authentic execution environments. Through trace-level interaction analysis, the authors isolate environment authenticity signals—such as browser profile, cookies, fingerprint stability—as the main differentiator between successful and failed bypasses by agents with otherwise indistinguishable behavior. This challenges assumptions about the fundamental security of behavioral defenses and highlights the execution environment as the critical attack surface moving forward.

Key findings

• Commercial Captcha-solving services achieve near-perfect success rates (≈100%) on challenge-based Captchas such as hCaptcha Easy, reCaptcha v2, and Cloudflare Turnstile managed mode, with costs as low as $0.10 to $5.00 per 1,000 solves (Table 2).
• These services perform poorly against behavioral scoring defenses like reCaptcha v3, with top solver success rates only reaching ~63% (BestCaptchaSolver) and many services failing entirely to produce valid tokens.
• Off-the-shelf LLM-based browser agents mostly fail to bypass challenge-based Captchas unless a dedicated solver module is integrated (e.g., SKYVERN's proprietary solver pipeline).
• LLM agents also fail on non-interactive defenses like reCaptcha v3 and Turnstile despite successfully completing required web interaction workflows, indicating their failure stems from environmental detectability rather than task comprehension.
• Two agents (Browser-Use and NanoBrowser) exhibit nearly indistinguishable behavioral footprints, but only NanoBrowser—operating within a native browser profile with persistent cookies and fingerprint stability—consistently bypasses defenses (Section 5.3).
• These results isolate execution-environment authenticity signals (persistent browser state, fingerprint consistency) as the primary factor enabling bypass of non-interactive bot defenses, rather than behavioral realism alone.
• Challenge-based defenses increasingly rely on outsourcing to third-party solvers, which commoditize attacks at negligible cost, undermining their security benefit.
• Non-interactive defenses today depend on environment-layer trust signals, which are vulnerable if attackers can replicate or control high-fidelity execution contexts.

Threat model

The adversary is an economic web attacker aiming to automate interactions on protected websites for purposes such as credential stuffing, scraping, or spam. They can access commercial Captcha solving APIs, deploy cloud- or self-hosted LLM-based browser agents in various configurations, and automate repeated attempts within resource constraints. The adversary lacks privileged access to server internals or signing keys and cannot directly manipulate trusted browser environment signals controlled by the defense.

Methodology — deep read

The study adopts a systematic measurement and evaluation approach to assess web bot management defenses against modern attack stacks.

1. Threat Model & Assumptions: The adversary is an economically motivated web attacker aiming to bypass bot management to perform automated workflows like login submissions or scraping. The attacker may use commercial Captcha solver APIs, run cloud-hosted or self-hosted LLM-based browser agents, and automate repeated trials under fixed budgets. The attacker does not possess privileged access but tries to evade detection relying on cost-effective external solving services or autonomous agents.

2. Data & Environment: The authors deploy a controlled public testbed consisting of seven web applications on dedicated subdomains, each protected by one of multiple bot management configurations: hCaptcha (Easy/Hard), reCaptcha v2 (Checkbox/Invisible), reCaptcha v3, and Cloudflare Turnstile (Managed/Invisible). These apps implement identical HTML forms but differ only by defense configuration. The sites are live for ~6 months prior to experiments to gather realistic background traffic and ensure behavioral detection systems are calibrated.

3. Architecture/Algorithm: For challenge-based defenses, third-party solver services are queried via their official APIs to obtain challenge solutions or tokens. Seven commercial solver platforms are evaluated, selected by user volume and ecosystem popularity. For LLM-based agents, six systems spanning cloud-hosted, self-hosted, AI-assisted browsers, and browser extension deployments are tested unmodified in default configurations. Agents receive standardized prompts to complete form submission workflows while being instrumented to record full browser interaction traces.

4. Training Regime: No training is performed by the authors. Instead, off-the-shelf solver services and pretrained LLM agents (some open-source, some commercial) are evaluated as-is. Agents thus operate under realistic default conditions rather than modified, optimized settings.

5. Evaluation Protocol: Each solver and agent attempts 100 trials per defense configuration. Success is defined as correctly bypassing the Captcha or behavioral challenge and completing the protected workflow submission. Metrics include success rate, average solving time, and interaction trace similarity. Behavioral realism is assessed via event trace comparison; environment authenticity is inferred from browser state parameters such as presence of cookies, stable fingerprinting, and execution environment.

6. Reproducibility: The authors provide detailed descriptions of deployed defenses, agent types, and solver services evaluated, but the datasets are publicly hosted web environments and proprietary solver APIs, so exact replication depends on service availability. Agent implementations are publicly accessible where possible. Detailed logs including browser interaction traces are collected.

Example end-to-end for third-party solvers: For reCaptcha v2 Checkbox challenges, the workflow programmatically submits the challenge images to solver service APIs, receives a solution token, injects it into the form, and submits. The token validity is verified via official vendor verification endpoints. Success rates and timings are recorded.

Example end-to-end for LLM agents: Agents receive natural-language prompts to complete a login form on protected sites. They autonomously perceive DOM elements, execute clicks and inputs, and attempt challenge solving if possible. Browsing sessions are recorded to analyze mouse movements, page events, and browser state. Behavioral similarity and environment signals are compared to understand bypass successes or failures.

Technical innovations

• Systematic side-by-side evaluation of commercial Captcha solvers and LLM-based browser agents against both interactive and non-interactive bot defenses deployed in controlled but publicly accessible environments.
• Fine-grained interaction trace analysis isolating execution environment authenticity signals—like browser profile continuity and fingerprint stability—as the key determinants of behavioral defense bypass rather than interaction behavior.
• Characterization of the solver ecosystem's operational cost-performance tradeoffs across challenge types, revealing commoditized near-perfect bypass for image-based Captchas but substantial struggle for behaviorally scored systems.
• Diverse LLM-browser-agent taxonomy including cloud-hosted, self-hosted, AI-assisted, and browser-extension configurations demonstrates how deployment modality impacts bot defense evasion capabilities.

Datasets

• Controlled web testbed: 7 HTML login form pages each protected by different bot defenses including hCaptcha (Easy/Hard), reCaptcha v2 (Checkbox/Invisible), reCaptcha v3, Cloudflare Turnstile (Managed/Invisible) — publicly accessible for ~6 months

Baselines vs proposed

• 2Captcha (reCaptcha v2 Checkbox): success rate = 100% vs proposed LLM agents generally fail (mostly ✗ except SKYVERN ✓ only with solver module)
• BestCaptchaSolver (reCaptcha v3): success rate = 63% vs LLM agents fail on reCaptcha v3 despite correct workflow execution
• AZCaptcha (hCaptcha Easy): success rate = 100% vs no LLM agents bypass hCaptcha without dedicated solver
• NanoBrowser (reCaptcha v3): success rate = high based on environment authenticity vs Browser-Use with behavioral parity but failure

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.18659.

Fig 1: LLM-based Browser Agents Workflow Diagram.

Fig 2: Sample Login Pages Protected with hCaptcha and

Fig 3 (page 3).

Fig 4 (page 3).

Fig 5 (page 3).

Fig 6 (page 3).

Fig 7 (page 3).

Fig 8 (page 4).

Limitations

• Evaluations focus on default unmodified configurations of LLM agents; performance under fine-tuned or adversarially optimized agents remains unexplored.
• Solver services tested may change over time; pricing and capabilities reflect snapshot as of 2026 and may not generalize.
• Environment authenticity signals are inferred but not exhaustively dissected; full root cause attribution of bypasses could benefit from deeper fingerprinting analysis.
• Study does not include adversarially hardened bot defenses or adaptive defense updates triggered by agent behaviors.
• Experiments mostly restricted to login form workflows; applicability to other high-stakes flows like payments or multi-factor authentication bypass is uncertain.
• Non-public proprietary solver service internal architectures limit insight into underlying vulnerability mechanisms.

Open questions / follow-ons

• How can bot management systems reliably validate environment authenticity signals under increasing attacker control of browser state and fingerprinting primitives?
• Can LLM-based agents be adversarially tuned or architected to evade environmental detection without relying on dedicated solver modules?
• What new environmental or behavioral telemetry features can fundamentally raise the cost of automation for advanced bot agents?
• How do large-scale, adaptive bot defenses perform under active, iterative attacks by LLM-powered automation that learn and evolve over time?

Why it matters for bot defense

For bot-defense practitioners, this work highlights the shifting attack surface introduced by LLM-based browser agents that combine autonomous reasoning with direct web interaction. Defensive reliance on challenge-based Captchas is increasingly undermined by commoditized low-cost solver services. Behavioral and trust-based non-interactive defenses offer stronger resistance but their security fundamentally depends on the authenticity of the browser execution environment rather than pure interaction patterns. This stresses the importance of robust environment attestation techniques and challenges the efficacy of purely behavior-driven risk scores.

CAPTCHA systems should consider enhanced integration with environment authenticity signals and continuous attestation to maintain security. Developers must acknowledge the limitations of challenge outsourcing and environment spoofing by sophisticated LLM agents, designing layered defenses spanning token verification, environment fingerprinting, and anomaly detection. Understanding that agent behavioral similarity alone is insufficient to ensure bypass also calls for improved metrics and evaluation frameworks for LLM-powered automations.

Cite

@article{arxiv2607_18659,
  title={ Broken Gates: Re-evaluating Web Bot Defenses in the Age of LLM Agents },
  author={ Behzad Ousat and Nikita Turkmen and Lalchandra Rampersaud and Dillan Bailey and Amin Kharraz },
  journal={arXiv preprint arXiv:2607.18659},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.18659}
}

Read the full paper

• arXiv:2607.18659 (abstract)
• arXiv:2607.18659 (PDF)