Scalable Behaviour Cloning on Browser Using via Skill Distillation

Source: arXiv:2606.32014 · Published 2026-06-30 · By Kaisen Yang, Zheng Jiang, Yuzhao Peng, Houde Qian, Boshi Zhang, Youjie Zheng et al.

TL;DR

This paper addresses the core bottleneck in building capable web browser agents: decision-making under incomplete information rather than low-level browser control. The authors propose BrowserBC, a scalable behavior cloning framework that transforms large-scale human browser interaction traces into reusable, natural-language "skill cards". These skills capture procedural knowledge including task intent, preconditions, execution steps, milestones, failure handling, and termination signals. The skills are consolidated into a skill graph to avoid redundant accumulation. At inference, the agent retrieves and grounds task-relevant skills as guidance, enabling more efficient, robust, and transferable browser automation without brittle replay of low-level actions.

Evaluations on challenging benchmarks spanning reproducible self-hosted sites (WebArena-Hard), real-world live sites (ClawBench), and cross-substrate tasks (OSWorld desktop) show that BrowserBC substantially improves task success rates—boosting from 60.5% to 81.4% on WebArena-Hard and more than doubling success on ClawBench (32.9% to 68.4%). The skill distillation enables cross-model reuse, reducing interaction steps by 27% on average while improving robustness to layout changes. This demonstrates the value of distilling implicit human decision priors as modular natural-language procedural skills to scale browser agents' competence efficiently and transferably.

Key findings

• BrowserBC improves overall WebArena-Hard task success from 60.5% to 81.4% (+20.9 absolute points; Fig 3a).
• On ClawBench (live websites), BrowserBC nearly doubles solved tasks from 32.9% to 68.4% (+35.5 absolute points; Table 2), showing robustness beyond memorization.
• BrowserBC reduces mean agent interaction steps per task on WebArena-Hard from 31.2 to 22.7 (-27.3%; Fig 3a).
• Skills distilled by Claude-Sonnet-4.6 transfer effectively to smaller executing agents, improving success rates by +20 to +24 points (Fig 3b).
• An always-follow skill execution policy underperforms the calibrated BrowserBC that allows agent overrides (77.5% vs 81.4% success; §4.3).
• Skill graph merges and organizes thousands of demonstrations into modular reusable skills, controlling skill library growth and improving retrieval relevance (§3.5).
• Distilled skill cards explicitly represent intent, preconditions, steps, milestones, termination criteria, and failure recovery, enabling procedural abstraction (§3.4).

Threat model

The adversary is a web or digital automation task environment with partial observability—agents cannot see the full site map, reliable entry points, or final task outcomes upfront. The agent has access only to current page rendering and limited feedback signals; it cannot directly manipulate server state beyond legitimate UI interactions. The threat is the complexity and volatility of web interfaces causing brittle behavior if agents act without procedural priors. The system assumes no malicious actors actively trying to deceive or confuse agents.

Methodology — deep read

The authors begin by framing the key challenge for web agents as high-level decision-making under incomplete information rather than low-level action execution. The adversary is a general digital automation task with partially observable browser states.

Data consists of large-scale human browser interaction trajectories (x, e1...eT, y) where each event includes observation, action, and feedback. These are noisy, long, and tied to transient page-specific artifacts.

The core algorithm converts raw trajectories into reusable procedural natural-language skills through a four-stage pipeline: (1) Behavioral Evidence Abstraction cleans traces by filtering irrelevant/noisy actions and segments them into meaningful subprocedures, extracting evidence units capturing instructions, contextual state before and after, behavioral summaries, feedback, and outcomes. (2) Procedural Skill Distillation employs an instructed language model (Claude-Sonnet-4.6) to convert evidence units into structured skill cards. Each skill card includes intent, applicability scope, preconditions, execution steps, progress milestones, terminal evidence, failure modes, recovery strategies, negative constraints, and provenance. This removes brittle page-specific details such as coordinates or DOM selectors.

(3) Skill Library Construction merges new distillations with existing skills when intents and structures are compatible or specializes them otherwise. Skills are organized in a directional skill graph with edges encoding temporal dependencies, specialization, or alternative procedures, enabling incremental consolidation and scalable growth.

(4) Skill-Conditioned Execution: At inference, a lightweight retriever selects a small set of relevant skills by semantic similarity and applies them as natural-language context guidance to the executing agent. The agent grounds skills dynamically against the current page rather than blindly replaying actions, monitoring for milestones, validation messages, and completion signals. This decouples distillation from execution so distilled skills can be reused across smaller or cheaper models.

Training uses Claude-Sonnet-4.6 for skill distillation, evaluated on three benchmarks: WebArena-Hard (258 human-verified tasks on six self-hosted sites with programmatic success checkers), ClawBench (152 tasks on live production websites with agentic evaluators), and OSWorld (30 desktop GUI tasks). The same executing agent architecture with and without BrowserBC skill context is compared in a controlled regime holding models and runtimes fixed.

Evaluation metrics include task success rate, steps per task, cross-site generalization, failure recovery, and skill library efficiency. Ablations explore decoupling distiller and executor model, skill execution policies (always follow vs calibrated), and cross-substrate applicability. The skill library is audited to remove answer leakage or explicit checker details.

Reproducibility is supported by published code and public benchmarks. Exact pretraining details of Claude-Sonnet-4.6 are not disclosed but the dataset of traces is sourced from multiple prior datasets and collected in-house. The skill distillation pipeline is deterministic given the model and data.

For a concrete end-to-end example, a raw trajectory of filling and submitting a web form is segmented into subprocedures, stripped of coordinates and session info, distilled into a skill card specifying intent (submit info), preconditions (form page loaded, known required fields), steps (locate fields by label, enter data, submit), milestone checks (validation messages), and failure recovery (missing required fields). This skill is retrieved at runtime on a new site with differing layout but similar forms, enabling the agent to fill the form robustly by following the distilled procedural knowledge rather than replaying brittle low-level actions.

Technical innovations

• Recasting web automation as skill-conditioned decision-making under partial observability rather than low-level action cloning.
• Distilling human browser trajectories into structured, natural-language skill cards capturing procedural knowledge (intent, preconditions, milestones, recovery) abstracted from brittle page-specific details.
• Constructing a scalable skill graph that merges, specializes, and organizes reusable skills to avoid library sprawl and improve retrieval efficiency.
• Decoupling skill distillation and execution models to enable reusable skill libraries that guide smaller, cheaper agents without retraining.
• Using natural-language skill representations as model-agnostic priors that the executing agent calibrates dynamically against live observations.

Datasets

• WebArena-Hard — 258 tasks — self-hosted web with programmatic task verifiers
• ClawBench — 152 tasks — live production websites with agentic evaluator
• OSWorld-style desktop tasks — 30 tasks — Ubuntu desktop GUI applications

Baselines vs proposed

• Base agent (no skills) on WebArena-Hard: success rate = 60.5%; BrowserBC: 81.4% (+20.9 pts)
• Base agent (no skills) on ClawBench: success rate = 32.9%; BrowserBC: 68.4% (+35.5 pts)
• Qwen-3.7-distilled skills with Sonnet executor: modest gains (~+7 pts); Sonnet-4.6-distilled skills with Sonnet executor: +24 pts (Fig 3b)
• Always-follow skill execution policy on WebArena-Hard: 77.5% success vs BrowserBC calibrated execution: 81.4%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.32014.

Fig 1: Motivation of BrowserBC. BrowserBC turns human browsing traces into reusable natural-language skills,

Fig 2: Overview of BROWSERBC. BrowserBC cleans and segments raw browser traces, distills them into

Fig 3 (page 2).

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 2).

Fig 7 (page 2).

Fig 8 (page 4).

Limitations

• Skill distillation depends heavily on the quality and coverage of human browser traces; rare or unseen tasks may not be supported.
• While promising, the current evaluation on desktop OSWorld tasks is limited in size (30 tasks) and does not demonstrate full cross-substrate generalization.
• The reliance on natural-language skills introduces some ambiguity and potential misinterpretation by executing agents, possibly limiting precision on highly dynamic or novel pages.
• Agentic evaluation on live ClawBench may introduce variability due to website volatility and nondeterministic state changes.
• No detailed adversarial robustness evaluation; exploits that manipulate page states or feedback to mislead skills are unaddressed.
• The skill distillation model (Claude-Sonnet-4.6) is a proprietary large language model limiting full community reproduction.

Open questions / follow-ons

• How effectively can the skill distillation approach scale to rare or novel tasks with limited demonstrations?
• Can skill graphs be efficiently updated online with active learning or real-time agent feedback to handle fast-changing websites?
• What mechanisms can improve skill execution precision and reduce ambiguity when natural language guidance conflicts with live page layouts?
• How robust are the distilled skills under adversarial manipulations or intentional page obfuscation?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, BrowserBC represents a shift from treating browser automation as low-level action replay to a high-level, decision-prior-driven problem. By distilling human browsing behavior into reusable procedural skills, agents can act more intelligently and robustly on unfamiliar or volatile web pages, reducing brittle trial-and-error interactions. This insight may inform domain adaptation or anomaly detection approaches that model typical human navigational patterns as composable skill graphs rather than surface-level inputs.

From a defense perspective, this also suggests that attackers leveraging such skill distillation may produce more human-like, context-aware automation that is harder to detect by superficial behavioral heuristics. Detecting automation that applies reusable natural-language procedural priors rather than rote patterns may require defenses to leverage higher-level semantic or intent-based signals. Thus, understanding how skill distillation frameworks operate can help shape more resilient detection and CAPTCHA challenge designs that hinge on procedural, contextual task comprehension rather than isolated interactions.

Cite

@article{arxiv2606_32014,
  title={ Scalable Behaviour Cloning on Browser Using via Skill Distillation },
  author={ Kaisen Yang and Zheng Jiang and Yuzhao Peng and Houde Qian and Boshi Zhang and Youjie Zheng and Shijin Hong and Qingle Liu and Ruoyu Han and Bohan Lyu and Bingxiang He and Eren Cai and Calvin Xiao and Qinhuai Na },
  journal={arXiv preprint arXiv:2606.32014},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.32014}
}

Read the full paper

• arXiv:2606.32014 (abstract)
• arXiv:2606.32014 (PDF)