Internal APIs Are All You Need: Shadow APIs, Shared Discovery, and the Case Against Browser-First Agent Architectures

Source: arXiv:2604.00694 · Published 2026-04-01 · By Lewis Tham, Nicholas Mac Gregor Garcia, Jungpil Hahn

TL;DR

This paper addresses a core inefficiency in autonomous web agents that currently rely heavily on browser automation to interact with websites designed for human users. Such browser-first approaches require repeatedly navigating page DOMs and reverse-engineering APIs on each interaction—an expensive, slow, and brittle process replicated by many agents independently. The authors observe that virtually all modern websites expose internal "shadow" APIs that power their UIs but are not publicly documented or directly accessible by agents. Their novel system, Unbrowse, transforms this scenario by passively learning and collectively maintaining a shared route graph indexing callable internal APIs ("skills") from real browsing traffic. Agents can then query this shared graph for direct API calls, bypassing browsers in many cases. Empirically, over 94 live domains and information retrieval tasks, Unbrowse achieves 3.6× mean and 5.4× median speedups compared to Playwright browser automation, with hot routes resolving in under 100 ms. A three-path execution model ensures agents fall back to local cache or browser rediscovery if needed, maintaining correctness and coverage. The paper also introduces a three-tier micropayment economic model via the x402 protocol to sustainably incentivize route discovery, maintenance, and site-owner participation. Overall, Unbrowse offers a scalable, high-performance alternative to brittle browser-first agent designs through shared shadow-API discovery and usage incentives.

Key findings

• In a live-web benchmark across 94 domains, Unbrowse reduces information-retrieval latency from 3,404 ms (Playwright browser automation) to 950 ms (warmed cached routes), a 3.6× mean speedup and 5.4× median speedup (Fig. 7).
• Well-cached routes complete in under 100 ms, enabling near real-time agent workflows with multi-site interaction.
• The shared route graph passively learns APIs from browsing traffic and serves cached routes via direct API calls, eliminating DOM parsing and LLM token use for UI state reasoning.
• Unbrowse applies a three-path execution fallback: (1) local cache, (2) shared graph route, (3) full browser rediscovery, ensuring robustness and self-correction.
• A novel delta-based attribution mechanism fairly distributes route installation fees to contributors proportional to their marginal contributions, incentivizing ongoing route maintenance.
• The economic adoption condition froute < crediscovery (total route fee less than browser rediscovery cost) holds empirically, validating the micropayment pricing model.
• Tier 3 graph query fees for semantic route lookup add $0.001-$0.005 per query, but remain negligible compared to multi-second browser rediscovery latencies.
• Continuous verification and freshness scoring maintain route quality amidst schema drift, with a background verification loop running every 6 hours to detect endpoint failures.

Threat model

The adversary is an autonomous agent or automated client aiming to access website functionality efficiently. The adversary does not control the website or its APIs but can observe network traffic when browsing or has access to the shared route graph. The agent cannot directly obtain private credentials from other agents or circumvent rate limiting and authentication enforced by sites. Site owners may optionally opt into execution fees to control access economically. The threat model excludes active adversarial API obfuscation or adversarial manipulation of route contributions—these are future challenges.

Methodology — deep read

1. Threat model & assumptions: The adversary is implicit—agents or automated clients seeking web access must either reverse-engineer site APIs through browser automation or consult the shared route graph. Attacker knowledge is limited to public sites, but direct private API metadata is not disclosed unless reverse-engineered. The system assumes that agents incur significant cost browsing UIs and that direct API discovery is preferable if cheaper and reliable. Site owners may optionally participate in the economic model.

2. Data: The system collects network traffic passively from real agent browsing sessions across 94 live web domains. This raw traffic includes requests/responses with noise (analytics, static assets) which the pipeline heuristically filters using Content-Type analysis (JSON/XML), URL patterns, request methods (POST/PUT/PATCH), and response structure. Extracted endpoints, parameters, response schemas, and auth metadata form route abstractions and capabilities. No synthetic or labeled datasets are mentioned; the learned dataset is dynamically collected.

3. Architecture / algorithm: Unbrowse consists of two layers: a Capability Layer for passive route discovery and maintenance, and an Economic Layer implementing micropayments and incentives. The Capability Layer observes browsing network traffic, filters noise, extracts and normalizes API routes, packages these into "skills" using the AgentSkills.io standard (including docs, auth patterns, typed client code). These skills are stored as a shared route graph index supporting semantic vector search. The orchestrator resolves agent queries through a priority chain (local cache, shared graph search with composite relevance and trust scoring, browser fallback). The Economic Layer manages a three-tier micropayment system using the x402 protocol: Tier 1 (one-time skill install payments), Tier 2 (optional per-execution site-owner fees), Tier 3 (per-query graph search fees). Delta-based contribution attribution assigns rewards proportional to route improvements.

4. Training regime: N/A as this is a system engineering and deployment paper without machine learning model training per se. Semantic vector embeddings for route representations are mentioned, but embedding training details are not disclosed and considered replaceable implementation choices.

5. Evaluation protocol: The authors benchmarked Unbrowse on 94 live web domains for equivalent information retrieval tasks comparing against Playwright browser automation. Metrics include end-to-end latency (mean, median), success rates, cache warm/cold start performance. The system’s three-path execution fallback and continuous verification monitor route reliability. No adversarial red-teaming or distribution shift evaluation beyond cold cache scenarios is detailed. Statistical significance tests are not reported.

6. Reproducibility: Unbrowse is released as an open-source npm package with CLI and local HTTP API. The browser capture runtime (Kuri) is also open source. The knowledge graph is cloud-hosted but code and interfaces for skill generation and local execution are publicly available. Exact dataset traces are not published, as data comes from real browsing traffic. Overall reproducibility is plausible but dependent on access to live web and real traffic.

Concrete example: When an agent requests a route for a given task, the orchestrator first queries the local cache (Path 1) for a nearby match. If missing, it queries the shared route graph (Path 2) using semantic search weighted by embedding similarity, route health, freshness, and verification signals. If a match is found, the corresponding skill package is downloaded after a one-time micropayment (Tier 1). The agent then executes API calls directly, bypassing the browser. If Path 2 fails, full browser automation (Path 3) launches headless Chrome via Kuri to interactively rediscover routes by capturing network requests, which are then published back to update the shared graph for future use.

Technical innovations

• A shared route graph architecture that passively learns and collectively maintains internal shadow API routes from real browsing traffic, rather than relying on manual submissions or official public APIs.
• A three-path execution model (local cache, shared graph, browser fallback) enabling agents to transparently trade off speed, accuracy, and trust, while preserving an 'exit option' to discipline the route market.
• A novel delta-based contribution attribution mechanism that quantifies marginal improvements to route quality and fairly distributes micropayment revenue among multiple contributors over time.
• A three-tier micropayment economic model via the x402 protocol, separating fees for skill installation, optional site-owner per-execution charges, and per-query graph lookups, grounded by an adoption condition comparing fees to browser rediscovery cost.
• Continuous automated verification and schema drift detection running locally every 6 hours to maintain route freshness and disable broken routes without human intervention.

Datasets

• Live browsing traffic from 94 web domains — dynamic real-world data collected passively during agent interactions — not publicly released

Baselines vs proposed

• Playwright browser automation: mean latency = 3,404 ms vs Unbrowse warmed cache execution: mean latency = 950 ms
• Median latency: Playwright = ~5,140 ms vs Unbrowse = ~950 ms (5.4× median speedup)
• Well-cached routes: execution time under 100 ms vs browser baseline multi-second scale

Limitations

• The approach depends on passive traffic capture by agents actually browsing sites; cold-start coverage on rarely visited or private sites may be low.
• No formal analysis of the incentive mechanism’s resistance to Sybil attacks or contributor manipulation, though a minimum-delta threshold is applied.
• The continuous verification mechanism currently runs locally and depends on safe GET requests; non-GET endpoints are less well monitored.
• Economic model is conceptual; full equilibrium and welfare analyses are deferred for future work.
• The evaluation focuses on latency and task success but lacks adversarial robustness evaluation or tests under active UI/API obfuscation.
• Dependency on the x402 micropayment protocol and opt-in participation might limit adoption where economic incentives are misaligned.

Open questions / follow-ons

• How robust is the delta-based contribution attribution against sybil or spam attacks in large-scale, real-world deployments?
• What are the equilibrium properties and welfare implications of the micropayment economic model in diverse agent ecosystems?
• How can automated verification be extended to non-GET and mutation endpoints without risking data integrity or authentication breaches?
• How might future web standards evolve to natively support agentic access, complementing or superseding passive discovery?

Why it matters for bot defense

This work highlights the inefficiencies of current browser-first agent designs, which are common targets for bot detection systems including CAPTCHA challenges. By enabling agents to bypass expensive UI navigation via direct internal API calls discovered and shared collectively, the system challenges assumptions about interaction patterns that bot defenders rely on. CAPTCHA practitioners should consider that advanced agents may leverage shadow APIs invisible to traditional browser-based defense heuristics, demanding more nuanced signals beyond browser events. Additionally, the shared route graph concept suggests new post-discovery access patterns with lower latency that bots can exploit, influencing risk modeling. However, the micropayment and opt-in models also create economic incentives for site owners and defenders to participate in controlled, accountable API access, potentially informing CAPTCHA design to balance usability and security in a multi-agent future.

Cite

@article{arxiv2604_00694,
  title={ Internal APIs Are All You Need: Shadow APIs, Shared Discovery, and the Case Against Browser-First Agent Architectures },
  author={ Lewis Tham and Nicholas Mac Gregor Garcia and Jungpil Hahn },
  journal={arXiv preprint arXiv:2604.00694},
  year={ 2026 },
  url={https://arxiv.org/abs/2604.00694}
}

Read the full paper

• arXiv:2604.00694 (abstract)
• arXiv:2604.00694 (PDF)