How Benchmarks Mis-Score Computer-Use Agents
Source: arXiv:2607.28367 · Published 2026-07-30 · By Zihan Dong, Zhiyuan Ma, Zekun Wang, Yunqing Li, Zirou Liu, Ruixuan Deng et al.
TL;DR
This paper examines the reliability of benchmarks used to evaluate Computer-Use Agents (CUA) that automate tasks on the web and desktop environments. The authors argue that current benchmarks rely on brittle scripted oracles for scoring, which leads to frequent mis-scoring of agent failure. They conceptualize the evaluation pipeline as four stages—task construction, execution environment, scoring, and reporting—and identify three main reliability threats: stale or invalid tasks/environments, evaluator fragility rejecting valid results or rewarding shortcuts, and diagnostic opacity from incomplete trajectory observation. An empirical audit of 150 failure trajectories from five major benchmarks finds that 15.3% of failure verdicts are erroneous, with 10.7% being evaluator false negatives and 4.7% broken tasks. Genuine agent failures are dominated by planning and verification/feedback errors rather than execution or grounding faults, revealing how a single scalar success rate obscures failure causes. The authors translate these insights into stage-specific design guidelines to improve the fidelity and interpretability of CUA evaluation. They also discuss how evolving task pools, environment variation, calibrated judge-based scoring, and richer diagnostic reporting can help mitigate current shortcomings.
Key findings
- 15.3% of FAIL verdicts across 5 CUA benchmarks audited are incorrect, including 10.7% evaluator false negatives and 4.7% broken tasks (Table 2).
- WebArena showed a 21.7% evaluator false negative rate, rejecting valid agent behavior due to brittle string matching.
- AssistantBench tasks suffered 21.7% broken failures from environment drift including dead search engines and CAPTCHA walls.
- Among genuine failures, 39.3% were Tier 3 verification and feedback failures (29.5% feedback-blind no-op repetitions), 35.2% were Tier 1 planning and specification failures, and only 13.9% were Tier 2 execution and grounding failures.
- Removing screenshots from trajectories reduced detection of evaluator false negatives, lowering Stage 1 agreement from kappa=0.71 to kappa=0.60.
- LLM judges and human annotators substantially agree on failure verdicts (kappa=0.71) but only moderately agree on failure diagnosis (kappa=0.41).
- Task validity issues arise from environment staleness and unsatisfiable tasks in released VM images or APIs.
- Reporting only single scalar success rates severely hides failure mode diversity, reducing actionable insights.
Threat model
The adversary is essentially the benchmark evaluation pipeline errors, which include stale or unsolvable tasks, environment drift invalidating outcomes, brittle evaluator oracles rejecting valid solutions, and incomplete trajectory observation obscuring failure causes. The adversary is not an external attacker but the risk of mis-attribution of agent capability from pipeline flaws. The evaluation assumes that agents operate legitimately under declared conditions without access to leaked answers beyond retrieval scopes.
Methodology — deep read
The authors conceptualize CUA benchmark scoring as a pipeline of four inductive stages: (1) task construction, generating domain-valid, specified goals and constraints; (2) execution environment, which must be reproducible and observable, including the OS image, application versions, and tool scaffolds; (3) scoring, where oracles, rubrics, or judges assess agent outcomes and trajectories; and (4) reporting, producing aggregate verdicts plus diagnostic details. They identify three reliability threats mapped to these stages: task validity and contamination (stage 1 and 2), evaluator fragility (stage 3), and observability and diagnostic opacity (stage 2 and 4).
For empirical validation, they audit 150 publicly available zero-reward (failure) trajectories sampled from five benchmarks: OSWorld, WebArena, VisualWebArena, WorkArena, and AssistantBench. The audit population was stratified by benchmark and agent and randomly sampled with deterministic seed. The data included full step-level reasoning, actions, screenshots, and observations.
Two vision-enabled LLM annotators (OpenAI GPT-5.5 Codex CLI v0.144.5 and Anthropic Claude Sonnet-5) independently labeled each failure trajectory in a two-stage labeling pipeline: Stage 1 classified a failure as genuine agent failure, evaluator false negative, broken task, or unclear based on trajectory completeness and evidence; Stage 2, only on genuine failures, assigned a failure diagnosis using a three-tier taxonomy adapted from the MAST framework, distinguishing planning/specification failures (tier 1), execution/grounding failures (tier 2), and verification/feedback failures (tier 3).
A subset of trajectories where LLMs disagreed was reviewed by two independent human groups blind to each other and the LLM outputs, enhancing label reliability via majority vote. The authors tracked inter-annotator agreement (Cohen's kappa and Fleiss' kappa) for verdict and diagnosis labels.
They evaluated the impact of evidence completeness by re-labeling with screenshot information removed, showing that missing visual cues impede correct evaluation and reduce agreement.
Finally, they analyzed failure frequencies, disagreement patterns, and diagnostic distributions across benchmarks and agent types to quantify evaluator errors versus genuine failures and characterize dominant error modes. The audit methodology carefully separates benchmark faults from agent faults, capturing concrete examples like a task with unsatisfiable VM constraints or legitimate alternative correct answers rejected by brittle oracles.
No new agent runs or code releases were produced. Analysis focused on trajectory review and meta-annotation of benchmark verdicts. The authors note that the evaluation is limited to released trajectories and does not cover false positives (incorrectly accepted successes).
Technical innovations
- Framing CUA benchmark scoring as a four-stage reliability pipeline: task construction, execution environment, scoring, and reporting.
- Introducing a diagnostic taxonomy adapted from MAST to classify genuine agent failures into planning, execution, and verification failure modes.
- Quantitative audit methodology combining vision-enabled LLM judges with human groups for label verification on failure trajectories from multiple benchmarks.
- Empirical demonstration of evaluator false negatives and broken tasks as prevalent error sources corrupting benchmark fail/fail verdicts.
Datasets
- OSWorld — 57 failure trajectories — publicly released with VM state checks
- WebArena — 23 failure trajectories — public benchmark with string, URL, and DOM checks
- VisualWebArena — 23 failure trajectories — public with visual inspections and programmatic checks
- WorkArena — 24 failure trajectories — ServiceNow form completion checks
- AssistantBench — 23 failure trajectories — open web answer matching
Baselines vs proposed
- WebArena evaluator false negative rate: 21.7% (vs proposed audit revealing evaluator brittleness and verifiable false negatives)
- AssistantBench broken tasks rate: 21.7% (vs ideal task validity assumptions)
- OSWorld evaluator false negative rate: 14.0% including unsatisfiable released environments
- WorkArena zero detected evaluator false negatives in this sample (0/24)
- Aggregate FAIL error rate across benchmarks: reported 15.3% wrong verdicts, audit-validated
Limitations
- Audit only reviews failure trajectories; false positives among success verdicts are not examined.
- The LLM judges used are from current API snapshots without exact reproducibility guarantees.
- Human diagnostic agreement is moderate (kappa=0.41), indicating some subjectivity or annotation complexity.
- The 150 trajectory sample is stratified but relatively small, possibly missing some rare error modes.
- Analysis focuses on publicly released benchmarks and trajectories, limiting generalization to private or proprietary datasets.
- No adversarial or distribution-shift evaluations are presented; robustness to environment drift is only qualitatively discussed.
Open questions / follow-ons
- How to design scalable and automated diagnostic probes that accurately flag planning and grounding failures without human judges?
- How can evolving or procedural task generation and living pools be standardized for continuous benchmark freshness?
- What are the best practices for calibrating Agent-as-a-Judge models to approach or exceed human label accuracy in evaluating open-ended, long-horizon CUA tasks?
- How do failure mode distributions change under adversarial or out-of-distribution environment shifts?
Why it matters for bot defense
Bot-defense and CAPTCHA system developers seeking to benchmark automated agents' ability to interact with web interfaces should be wary of brittle scripted oracles that mis-score genuine agent success or failure. The paper’s reliability framework underscores the importance of task validity, environment stability, evaluator robustness, and full trajectory observability, which align with challenges in reliably detecting automated human-like behavior in adversarial settings. For robust bot detection, one should design evaluation and detection pipelines that explicitly consider task drift, partial success alternatives, and use multi-dimensional diagnostic signals rather than scalar success/failure labels. Furthermore, releasing diagnostic evidence like screenshots and timestamps enables forensic analysis that can help separate true bot failures from evaluator or environment errors—a critical capability in high-stakes CAPTCHA evaluation where false positives and negatives have user experience consequences.
Cite
@article{arxiv2607_28367,
title={ How Benchmarks Mis-Score Computer-Use Agents },
author={ Zihan Dong and Zhiyuan Ma and Zekun Wang and Yunqing Li and Zirou Liu and Ruixuan Deng and Qishi Zhan and Rui Qian },
journal={arXiv preprint arXiv:2607.28367},
year={ 2026 },
url={https://arxiv.org/abs/2607.28367}
}