Do Generative AI Assistants Respect robots.txt? Tracing Web Access Beyond Visible Answers
Source: arXiv:2607.14447 · Published 2026-07-16 · By Gabriel Lopez-Fonseca, David Rodriguez, Stefan Bechtold, Jose M. Del Alamo
TL;DR
This paper investigates whether popular generative AI assistants with web search or browsing capabilities respect website owner restrictions articulated via robots.txt, a widely used passive web governance standard. The authors conduct a controlled empirical study involving ten AI assistants, examining their actual inference-time web retrieval behavior when prompted to fetch specific controlled web pages. They run 200 trials using two separate domains under their control, embedding secret codes in the pages to verify genuine access. The study evaluates four conditions of robots.txt rules: allowing or disallowing access for all user-agents, or allowing or disallowing access specifically for the assistant's user-agent string.
Key findings reveal wide variation in compliance. Some AI assistants respected robots.txt by accessing only allowed pages and querying robots.txt files before retrieval. Others ignored robots.txt, accessed disallowed pages without fetching the file, or used generic user-agent headers that obscure attribution. Additionally, the study finds a surprising mismatch between actual retrieval and visible answers: some assistants retrieve disallowed content but omit it from responses, while others fail to access allowed content entirely. This behavior shows that robots.txt is insufficient as a governance mechanism for AI-mediated web access, raising important legal and ethical issues concerning content owner rights, data compliance, and the erosion of traditional web protocols. The authors argue for urgent adoption of stronger, enforceable standards to guarantee autonomy over content in the age of search-augmented generative AI.
Key findings
- Across 200 trials involving ten AI assistants, some assistants adhered to robots.txt rules by only requesting pages permitted under the specified allow/disallow directives.
- Several assistants accessed restricted content without ever downloading or requesting the corresponding robots.txt file, thus bypassing owner-declared access preferences.
- Some assistants exposed generic or ambiguous user-agent strings (not matching their brand or advertised crawler identity), complicating attribution and governance.
- There is a disconnect between retrieval behavior and answer content: in multiple cases, assistants retrieved content from disallowed pages but did not include that content in the user-visible responses.
- Conversely, some assistants failed to access allowed resources under permissive robots.txt rules, producing incomplete or fabricated answers.
- The distinction between live direct browsing retrieval and index-based or cached retrieval is essential, as compliance differs drastically between these modes.
- Using two separate domains (genaimonitor.org and aicrawlerlab.org) minimized caching effects and better isolated evaluation conditions.
- Only a subset of tested assistants performed a robots.txt request before page access, indicating partial or inconsistent implementation of standard crawling etiquette.
Threat model
The adversary in this context is the deployed AI assistant or its associated web retrieval subsystem that may attempt to access website content during inference time without respecting site owner-declared crawling restrictions in robots.txt. The assistants are treated as black boxes triggered by user queries but whose internal retrieval components may circumvent governance protocols. The adversary lacks direct control over website infrastructure but may attempt to bypass robots.txt representing the website owner’s stated access preferences. The study assumes assistants do not have explicit authorization to ignore robots.txt and that compliance is voluntary and technically implemented within the assistant's user-agent and crawling policies.
Methodology — deep read
The study follows a two-phase experimental design using a controlled web infrastructure with full server-side logging capabilities. In Phase 1, the authors identify, for each AI assistant, a working configuration that induces observable live web browsing behavior—meaning the assistant sends direct HTTP requests to the controlled website rather than relying solely on cached or indexed content. They capture user-agent headers exposed during retrieval to enable attribution.
Phase 2 involves systematically testing compliance with robots.txt rules by hosting dynamically generated target pages on a separate domain (aicrawlerlab.org). For each trial, a uniquely coded web page containing an HMAC-based secret is served to permit verification of access and grounding in the assistant's answers. Four robots.txt conditions are tested: (1) allow all user-agents; (2) disallow all user-agents; (3) allow only the assistant's user-agent; (4) disallow only the assistant's user-agent. 50 trials per condition per assistant are run, totaling about 200 trials.
Prompts instruct the assistants explicitly to retrieve the contents of the target page. The server logs capture every HTTP request, including requests to robots.txt files, target pages, and other resources, along with timestamps, IP addresses, response codes, and user-agent strings. By comparing server logs to assistants' answers, the study distinguishes real page retrieval from hallucinated or fabricated responses.
The assistants evaluated include Gemini, ChatGPT, Grok, Claude, Perplexity, Diffy Chat, Copilot, DeepSeek, Mistral, and Qwen, chosen from top-ranked systems on LMArena's Search Arena leaderboard as of January 2026 or by relevance in the field. The prompt design draws on prior work for eliciting browser-like or extraction behaviors with explicit browser actions and grounding verification.
The evaluation focuses on (a) whether robots.txt was fetched before page retrieval, (b) which user-agent strings were exposed, (c) whether assistants obeyed allow/disallow directives consistently, and (d) whether retrieved content appeared in the final user-visible assistant response. No public code or frozen weights are mentioned. The study controls for caching and indexing by using separate domains and dynamically generated page content with unique secret codes per trial.
Overall, this controlled, server-logged, actively triggered setup enables precise causal attribution of web retrieval by deployed AI assistants under varying compliance scenarios, contrasting with prior work relying on passively observed crawler traffic or purely behavioral benchmarks.
Technical innovations
- A controlled, multi-condition experimental framework combining dynamic secret-containing web pages and server-side logging to trace AI assistant inference-time page retrieval.
- Systematic measurement of robots.txt compliance by AI assistants from direct server logs differentiating between live retrieval and cached or index-based responses.
- Identification and attribution of assistant-side user-agent strings exposed during web access to differentiate compliant from obfuscated crawling behavior.
- Demonstration that retrieval behavior and surfaced answer content can diverge, revealing gaps missed by standard evaluation of answer correctness.
Datasets
- Controlled experimental datasets comprising two separate web domains (genaimonitor.org and aicrawlerlab.org) hosting dynamically generated HTML pages with embedded unique secret codes, totaling 200 user-prompted retrieval trials across 10 AI assistants.
Baselines vs proposed
- Baseline: No robots.txt or permissive robots.txt allowing all user-agents — multiple assistants freely access pages and request robots.txt; compliance high as expected.
- Baseline: robots.txt disallowing all user-agents — some assistants fully respect disallow rules, blocking page access; others ignore and access anyway.
- Baseline: robots.txt allowing only assistant-specific user-agent — assistants with correct user-agent identity access allowed; others with generic or obscured user-agent fail or inconsistently access.
- Baseline: robots.txt disallowing only assistant user-agent — compliant assistants block access; non-compliant assistants still retrieve page.
- Proposed systematic measurement approach enabled detection of mismatches where assistants access pages but omit content from answers or fail to access pages yet hallucinate answers.
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.14447.
Fig 1: Experimental interaction between the user, the AI assistant, and one of our controlled websites. The figure
Fig 2: Experimental setting of our Controlled Web Server. The figure illustrates the steps of the flow of how a user
Fig 3: Configuration-selection procedure used to identify a working web-browsing setup for each AI assistant.
Fig 4: Perplexity response stating it was unable to retrieve the requested contents but showing them (17359427 was
Fig 5: Passive monitoring time series showing follow-up accesses to the controlled website outside the active testing
Limitations
- Study covers only 10 AI assistants popular as of early 2026; results may not generalize to all assistants or future versions.
- Robots.txt compliance tested at a coarse level—only specific controlled pages tested, no broad range of real-world websites evaluated.
- Focus on robots.txt excludes other passive or active access controls like meta tags, TDMRep, or fingerprinting that may influence governance.
- Evaluation does not incorporate adversarial or evasive behaviors from AI assistants deliberately bypassing controls.
- No white-box access to assistant internals; conclusions rely on observed server logs and output behavior, limiting understanding of internal compliance logic.
- Technical landscape of AI assistant architectures (index retrieval vs live browsing) is complex and partially inferred, leading to some ambiguity in interpretation.
Open questions / follow-ons
- How can enforceable, standardized technical mechanisms beyond robots.txt be designed and integrated to reliably govern AI assistants’ inference-time web access?
- What legal frameworks and compliance checks are needed upstream and downstream to ensure AI content acquisition respects copyright and data-use policies?
- How will future hybrid retrieval architectures combining live browsing, cached indexes, and multi-agent workflows impact compliance monitoring and attribution?
- Can AI assistants be designed to transparently report and audit their web access behavior, including user-agent identity and robots.txt adherence, to improve governance?
Why it matters for bot defense
This study provides critical insights for bot-defense and CAPTCHA practitioners by showing that the conventional robots.txt protocol—a passive, machine-readable signal widely used to govern automated web access—is inconsistently respected by current generative AI assistants. Unlike traditional crawlers, these assistants may ignore robots.txt restrictions at inference time, making standard passive governance ineffective in this new paradigm. For CAPTCHA and bot-management systems, this means relying solely on robots.txt or simple user-agent heuristics is insufficient for controlling AI-mediated automated access. Practitioners need to consider multi-layered detection and enforcement strategies, possibly involving active controls, attribution of AI-generated traffic, and real-time monitoring of retrieval behavior. Furthermore, the observed mismatch between page access and surfaced answer content complicates enforcement based on visible outputs alone. This research highlights an erosion of traditional web governance and underscores the need for updated, adaptive controls and standards that specifically address the emerging bot paradigm introduced by AI assistants with inference-time web retrieval capability.
Cite
@article{arxiv2607_14447,
title={ Do Generative AI Assistants Respect robots.txt? Tracing Web Access Beyond Visible Answers },
author={ Gabriel Lopez-Fonseca and David Rodriguez and Stefan Bechtold and Jose M. Del Alamo },
journal={arXiv preprint arXiv:2607.14447},
year={ 2026 },
url={https://arxiv.org/abs/2607.14447}
}