The Agentic Web Requires New Normative Infrastructure

Source: arXiv:2606.10711 · Published 2026-06-09 · By Cameron Pattison, Matthew Boulos, Noam Kolt, Changbai Li, Tiziano Piccardi, Seth Lazar

TL;DR

This paper addresses the emerging reality of the "agentic web," where users interact with online services predominantly through AI agents acting on their behalf. While technical advances have recently made powerful user-authorized AI agents feasible — capable of browsing, negotiating, transacting, and coordinating autonomously across web platforms — existing legal, normative, and platform policies do not distinguish these agents from malicious bots. As a result, platforms often indiscriminately block or degrade agent access, hindering the significant consumer and societal benefits properly delegated agents could provide. The authors argue that the primary bottlenecks to the agentic web’s promise are normative, not technical: outdated laws, non-transparent platform practices, and terms of service that afford platforms broad discretion to restrict AI agents without accountability.

The paper’s core contribution is a normative framework with three foundational principles (delegation, transparency, and proportional restriction) that together prescribe how to balance user agency and platform interests in governing AI agent access. They advocate for a societal dialogue and light-touch regulation enabling users to delegate authority to agents who identify themselves transparently, are restricted only to prevent concrete harms, and whose legitimate access mirrors the rights of their human principals. This normative infrastructure could unlock a more open, user-empowering agentic web and mitigate the ongoing arms race and legal conflicts between platforms and agent developers. The authors propose feasible regulatory interventions, particularly leveraging FTC authority, to begin enforcing these principles in practice.

Key findings

• Existing web access controls (robots.txt, fingerprinting, rate limits) are largely ineffective against modern AI agents that use sophisticated browsing behaviors, authenticated cookies, and browser automation frameworks like Selenium and Chromium-based browsers.
• Legal precedents (CFAA, terms of service bans, trespass to chattels) were designed for traditional web scraping bots rather than user-authorized AI agents, leading to overbroad platform blocking.
• AI agents differ substantially from earlier bots: they can act as delegated authorities, negotiate, transact, adapt over time, and pursue user-beneficial goals rather than just scraping data indiscriminately.
• The authors propose three principles: Delegation (users entitled to access should delegate it to agents), Transparency (platforms and agents disclose agent identity/access policies), and Proportional Restriction (platform limits must be narrowly targeted and least restrictive to concrete harms).
• Current platform practices include covert throttling and blocking of AI agents, often without disclosure, undermining user autonomy and market discipline.
• Legal cases such as Amazon.com Services LLC v. Perplexity AI show courts grappling with whether AI agents need both user and platform authorization, hinting at developing conjunctive standards.
• The proposed normative framework can be operationalized with extensions of existing tech like OAuth 2.0, OpenID Connect, W3C Verifiable Credentials for authenticated delegation and identification.
• A narrow but actionable regulatory path exists via FTC enforcement focusing on deception and unfair practices regarding covert agent blocking, complemented by state and federal legislative interventions targeting interoperability and platform power.

Threat model

The adversary is primarily large online platforms and cloud providers that have near-total discretion to block or throttle AI agent access under current terms of service and legal frameworks. These platforms may block agents indiscriminately due to technical inability or unwillingness to distinguish user-authorized agents from malicious bots, or to protect competitive advantage. The adversary does not include individual attackers, but rather intermediaries controlling access rights and technical gatekeeping. They cannot, under the proposed framework, block authorized agents without facing transparency and proportionality requirements.

Methodology — deep read

1. Threat Model and Assumptions: The adversary mostly concerns online platforms and intermediaries who currently treat all automated traffic, including user-authorized AI agents, as malicious or unauthorized bots. Platforms exercise unilateral control over access and deploy technical and legal means to block or degrade agent access. The paper assumes users want to delegate authority to AI agents to act legally and beneficially on their behalf, but platforms may act either to protect resources or to maintain competitive advantage.

2. Data: The paper does not present an empirical dataset or benchmark study but draws on a wide range of recent industry reports, technical descriptions, and legal cases relating to AI agent web access, platform blocking tactics, and associated litigation (e.g., Perplexity AI lawsuits, Cloudflare Turnstile adoption).

3. Architecture / Algorithm: Not applicable as this is primarily a normative and conceptual analysis. However, the technical feasibility of agent authentication and identification is discussed with references to OAuth 2.0, OpenID Connect, W3C Verifiable Credentials, and Decentralized Identifiers (DIDs), which could link user principals to AI agents in a verifiable, auditable manner.

4. Training Regime: Not relevant; no machine learning models are proposed or trained.

5. Evaluation Protocol: The paper evaluates concepts through critical analysis of current technical defenses (robots.txt, fingerprinting), legal precedents (CFAA, trespass, terms of service enforcement), and their unsuitability for AI agents. It also reviews case law and regulatory frameworks to assess feasibility of proposed principles. Figures illustrate the normative triad and regulatory path.

6. Reproducibility: Not applicable. The work is conceptual and normative, with no software or datasets released. The authors cite publicly available standards and ongoing developments around agent authentication and regulatory filings.

A concrete example walkthrough: For AI agents accessing paywalled content via user credentials, current platforms may treat such traffic as unauthorized scraping. Under the proposed framework, if the user authorizes the agent, and the agent identifies itself transparently using OAuth extensions, platforms would be obliged to permit access unless concrete harm could be demonstrated and addressed by the least restrictive means (e.g., usage quotas). This shifts the access paradigm from one based on technical and contractual blocking to principled, transparent delegation and accountability.

Technical innovations

• Articulation of a normative triad (delegation, transparency, proportional restriction) specifically tailored for AI agents acting on behalf of users online.
• Extension of traditional agency law concepts to digital AI agents, framing agents as legally delegated actors rather than independent unauthorized bots.
• Proposal to operationalize AI agent authentication and identification through existing standards like OAuth 2.0, OpenID Connect, and decentralized identity frameworks to enable verifiable delegation.
• Identification of legitimate versus illegitimate grounds for platform restrictions, distinguishing harms-based restrictions from anti-competitive or lock-in motivated blocking.
• Framing covert platform blocking as deceptive practice that could fall under FTC enforcement priorities, providing a novel regulatory foothold.

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.10711.

Fig 1: A normative triad for the agentic web. The user delegates authority to an agent (DELEGATION); the agent

Fig 2 (page 2).

Fig 3 (page 2).

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 2).

Limitations

• The paper is primarily normative and conceptual with no empirical measurements or quantitative evaluation of proposed principles in practice.
• No adversarial robustness analysis is conducted on the proposed authentication or transparency mechanisms.
• Ongoing legal cases and regulatory developments could evolve, making some current legal interpretations provisional or jurisdiction-specific.
• The authors focus largely on the US legal and regulatory environment, which may limit applicability in other international contexts with stronger data privacy and digital rights frameworks.
• Potential challenges around privacy, anonymity, and agent identity disclosure are noted but not fully resolved, especially in contexts requiring anonymous user access.
• Implementation hurdles remain for integrating agent identity and access management standards widely across heterogeneous online platforms.

Open questions / follow-ons

• How can agent identity and delegation protocols balance user privacy and anonymity with the transparency needed for platform trust?
• What technical standards and APIs are needed to broadly implement interoperable, verifiable agent authentication at internet scale?
• How would multi-agent interactions and emergent behaviors be governed under the normative infrastructure proposed?
• What are the economic models that could sustain the free and open agentic web, given challenges to advertising-driven revenue by agent-based content consumption?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this paper highlights a fundamental shift away from treating all automated access as malicious toward recognizing some AI agents as legitimately delegated user proxies. Traditional CAPTCHA mechanisms and automated traffic detection approaches risk conflating user-authorized agents with hostile bots, potentially blocking valuable user autonomy and functionality. Bot-defense engineers will need to design new systems that incorporate agent identity verification and support transparency about delegated access rights, possibly integrating with identity standards discussed (OAuth 2.0, OpenID Connect). Moreover, proportional restriction suggests limiting defenses to narrowly targeted harms rather than broad automated traffic blocks. This implies CAPTCHAs and bot challenges may need more granular, context-aware enforcement calibrated to distinguish authorized AI agents acting within delegated bounds from unauthorized bots or abusive traffic. In sum, this normative framework calls for rethinking bot detection and CAPTCHA gating to support an open, agentic web aligned with users’ delegated autonomy.

Cite

@article{arxiv2606_10711,
  title={ The Agentic Web Requires New Normative Infrastructure },
  author={ Cameron Pattison and Matthew Boulos and Noam Kolt and Changbai Li and Tiziano Piccardi and Seth Lazar },
  journal={arXiv preprint arXiv:2606.10711},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.10711}
}

Read the full paper

• arXiv:2606.10711 (abstract)
• arXiv:2606.10711 (PDF)