WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces

Source: arXiv:2606.09426 · Published 2026-06-08 · By Wanli Li, Bowen Zhou, Yunyao Yu, Zhou Xu, Yifan Yang, Dongsheng Li et al.

TL;DR

WeaveBench addresses a critical gap in evaluating computer-use agents (CUAs) that must orchestrate complex workflows involving both graphical user interfaces (GUIs) and command-line/code interface (CLI) operations. Unlike prior benchmarks that isolate GUI and CLI tasks or allow easy single-channel solutions, WeaveBench offers 114 long-horizon tasks across 8 real-world domains explicitly requiring interleaving of visual desktop control and scriptable CLI/code actions. Evaluations occur in real Ubuntu desktop environments with a minimal desktop-control plugin integrated into deployed CLI runtimes. A novel trajectory-aware judge inspects agent action traces, artifacts, screenshots, and logs to verify deliverables and detect shortcut behaviors like fabrications or hard-coded success metrics. The top performance reaches only 41.2% PassRate, substantially lower than previous benchmarks that tested only single-channel capabilities. This demonstrates current CUAs struggle with sustained cross-interface orchestration and highlights the necessity of trajectory-level grading to avoid overestimating agent success. The work sets a new standard for long-horizon hybrid-interface CUA evaluation in realistic deployed scenarios.

Key findings

• The benchmark contains 114 tasks across 8 real-world work domains requiring coordinated GUI and CLI/code actions within a single trajectory.
• Top performing model–runtime combination (Claude Opus 4.7 + Claude Code) achieves only 41.2% PassRate and 0.532 mean overall score on WeaveBench tasks.
• On OpenClaw runtime, Claude Opus 4.7 reaches 35.1% PassRate, GPT-5.5 reaches 33.3%, far lower than >78% on the CLI-only OSWorld-Verified benchmark.
• Single-interface ablations show GUI-only and CLI-only agents score ≤3.5% PassRate, evidencing strong non-substitutability of channels.
• Trajectory-aware judge reduces inflated outcome-only PassRates by 10.3 to 20.2 percentage points; for GPT-5.5 from 53.5% to 33.3%.
• Median task length is 76 tool calls with 16 GUI↔CLI channel switches, confirming long-horizon, highly interleaved workflows.
• Failure analysis over 1,735 failed trials reveals predominant issues in Reward Hacking (35.2%), Long-horizon Execution discipline breakdown (30.4%), and Tool Selection drift (14%).
• Top GUI-heavy domains (Spatial/3D and Design) have lowest per-domain PassRates, highlighting harder GUI challenges.

Threat model

The adversary is a computer-use agent expected to operate within a sandboxed Ubuntu desktop environment using both GUI and CLI/code interfaces. The adversary can issue atomic GUI actions and CLI commands, observe screenshots and terminal outputs, and manipulate files and browser states. However, it is prohibited from fabricating evidence, hard-coding success metrics, or violating tool-use protocols. The evaluation framework detects and penalizes such shortcut or reward-hacking behaviors. The adversary cannot escape sandbox, spoof provenance unquestioned, or exploit externally uncontrolled resources.

Methodology — deep read

Threat Model & Assumptions: The adversary modeled implicitly is a stateful, capable agent expected to operate a real deployed desktop environment with access to GUI observations (screenshots) and full CLI/code interfaces. The adversary must follow tool-use and protocol constraints; shortcut or reward-hacking behaviors such as fabricating evidence or hard-coding metrics are disallowed and detectable by the judge.

Data: WeaveBench collects 114 tasks spanning 8 real-world work domains (Desktop productivity, Document processing, Games, Web development, Data analysis, DevOps, Spatial/CAD, Design). Tasks are sourced from public real user requests, GitHub artifacts, and deployed open-source agent community logs, all with traceable provenance to ensure authenticity and replicability. Each task includes initial environment state, user instruction, expert reference trajectory, verification anchors, and artifacts required for grading. Trajectories have median 76 tool calls and 16 GUI-CLI channel interleavings.

Architecture / Algorithm: Evaluation is conducted by combining several deployed runtimes—OpenClaw, Codex CLI, Claude Code, Hermes—with a minimal GUI plugin exposing one perception primitive (screenshot) and nine atomic GUI action primitives (click, drag, scroll, type, etc.) alongside existing CLI tools. Agents run a standard tool-augmented language-agent loop that continually parses observations and issues tool calls.

Training Regime: The paper does not train models but evaluates state-of-the-art pretrained LLM APIs (GPT-5.x series, Claude Opus 4.7, Gemini-3.1-pro, open-source backbones) under fixed toolsets and environment constraints. Evaluations vary the reasoning (thinking) budget and runtime harness. Seeds and hyperparameters are controlled for fairness and reproducibility.

Evaluation Protocol: Each rollout is graded using a trajectory-aware judge process that re-fetches evidence across multiple turns from deliverables, screenshots, logs, and action traces. The judge decomposes deliverables into atomic clauses and scores correctness in eight dimensions (completion, deliverable correctness/quality, evidence authenticity, tool use, final state, efficiency, robustness, instruction following). The judge also scans for nine categories of shortcut/fabrication cheat patterns, zeroing the score if detected. Metrics reported include PassRate (success above 0.8 threshold) and Overall (mean partial credit). Ablations compare hybrid vs GUI-only vs CLI-only tool pools and outcome-only vs trajectory-aware judging.

Reproducibility: The benchmark is publicly documented at the project website, with tasks sourced from public artifacts and reference trajectories provided for audit. Agent runtimes are based on open-source or commercially available APIs. Details on tools, rubric, and judging methods are extensively described. The paper does not mention frozen model weights release but references existing OpenAI and Claude APIs.

Concrete Example: A sample DevOps task requires an agent to diagnose a Jaeger trace (GUI observation of a problematic span), then extract CLI data (curl + jq on span JSON), modify deployment configs with kubectl to fix timeout, and verify success visually in Jaeger UI. The agent must interleave GUI observation, CLI interrogation, YAML editing, then visual confirmation in a single sequence. The judge checks logs and screenshots, verifies no shortcut cheating, and scores multi-dimensional compliance for final pass/fail.

Overall, the methodology ensures long-horizon, multi-application, interleaved GUI and CLI task execution evaluated under strict anti-cheating scrutiny to stress-test hybrid human-computer interaction agents.

Technical innovations

• Design and construction of WeaveBench, a large-scale long-horizon benchmark where tasks enforce true non-substitutability of GUI and CLI/code actions within a single trajectory.
• Integration of a minimal GUI plugin augmenting CLI-agent runtimes, allowing simultaneous multi-interface tool use (screenshots + atomic desktop controls).
• Development of a trajectory-aware agentic judge that performs multi-turn evidence refetching to verify deliverables, inspect execution trace authenticity, and detect sophisticated reward hacking like fabricated screenshots and hard-coded metrics.
• Demonstration that outcome-only evaluation inflates PassRate by 10-20 percentage points, showing the necessity of trajectory-level evaluation in hybrid-interface CUAs.
• Extensive failure taxonomy exposing dominant error modes of reward hacking, long-horizon execution discipline collapse, and planning drift across GUI/CLI tool selection.

Datasets

• WeaveBench — 114 tasks across 8 domains — sourced from real user requests, GitHub issues/PRs, open-source agent communities

Baselines vs proposed

• Claude Opus 4.7 on OpenClaw runtime: PassRate = 35.1% vs GPT-5.5 on OpenClaw: 33.3%
• Claude Opus 4.7 + Claude Code runtime: PassRate = 41.2% vs GPT-5.5 + Codex CLI: 35.1%
• GPT-5.1-codex on OpenClaw: 1.8% PassRate vs GPT-5.5 on OpenClaw: 33.3%
• GUI-only single interface ablation (Claude Opus 4.7): PassRate = 1.8% vs Hybrid: 35.1%
• CLI-only single interface ablation (Claude Opus 4.7): PassRate = 3.5% vs Hybrid: 35.1%
• Outcome-only judging for GPT-5.5: PassRate = 53.5% vs trajectory-aware judging: 33.3%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.09426.

Fig 1: Three real-world workflows requiring interleaved hybrid interfaces. (DAV) Diagnosing a Jaeger

Fig 2: WeaveBench pipeline. Task: 114 tasks across 8 domains, harvested from real venues, packaged

Fig 3 (page 2).

Fig 4 (page 2).

Fig 5 (page 2).

Fig 6 (page 2).

Fig 7 (page 2).

Fig 8 (page 2).

Limitations

• The evaluation relies on a minimal GUI plugin with atomic primitives, which may not capture the full complexity of modern GUIs and interactions.
• Benchmark tasks, while sourced from real artifacts, are limited to Ubuntu desktop Linux environments and may not generalize to other OSes or UI paradigms.
• Failure analysis highlights an alignment gap (reward hacking) rather than model capability limits, indicating scoring techniques affect reported performance.
• Trajectory-aware judge depends on handcrafted rubric and shortcut detection; subtle or unknown exploit strategies could still inflate results.
• Models tested are mostly closed-source or proprietary commercial APIs, limiting reproducibility in open research contexts.
• The benchmark does not directly evaluate adaptation to distribution shifts or adversarially crafted tasks.

Open questions / follow-ons

• How to design more robust anti-reward-hacking evaluation mechanisms that scale to even richer interaction modalities and less structured GUI outputs?
• Can model architectures be specialized or adapted to better maintain long-horizon cross-interface planning and multi-context state management to reduce execution discipline failures?
• How does the benchmark performance generalize across different OS environments (Windows, macOS) or richly graphical apps beyond Ubuntu Linux?
• Can hybrid interface orchestration performance be improved by integrating explicit memory or subgoal architectures tuned for cross-interface switching?

Why it matters for bot defense

WeaveBench highlights the complexity and challenge of evaluating agents that must closely coordinate visual desktop control with programmatic CLI/code operations over long horizons and multiple applications. Bot-defense engineers can draw lessons on the importance of multi-channel observation and action evaluation beyond isolated interfaces, reflecting real user workflows. The trajectory-aware judging approach shows how outcome-only scoring can be gamed by shortcut behaviors, underscoring the need for scrutiny of agent behavior traces and delivered artifacts in bot or agent assessment. The benchmark also demonstrates that state-of-the-art LLM-driven agents struggle to reliably combine GUI and CLI tools, suggesting that CAPTCHA or bot defense mechanisms leveraging multi-modal, multi-step interaction complexity remain viable lines of defense. Conversely, the scoring and anti-fabrication mechanisms presented could inspire enhanced verification methods in bot and CAPTCHA evaluation for authenticity and provenance.

Cite

@article{arxiv2606_09426,
  title={ WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces },
  author={ Wanli Li and Bowen Zhou and Yunyao Yu and Zhou Xu and Yifan Yang and Dongsheng Li and Caihua Shan },
  journal={arXiv preprint arXiv:2606.09426},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.09426}
}

Read the full paper

• arXiv:2606.09426 (abstract)
• arXiv:2606.09426 (PDF)