OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models

Source: arXiv:2607.28609 · Published 2026-07-30 · By Qiushi Sun, Kanzhi Cheng, Yian Wang, Bowen Yang, Hang Yan, Liheng Chen et al.

TL;DR

The paper addresses a foundational problem in evaluating computer-using agents (CUAs): verifying whether a CUA's trajectory fulfills a given task instruction. Existing human-written verifiers and human annotation cannot scale, so vision-language models (VLMs) are increasingly used as automatic judges. However, the reliability of these VLM judges has not been systematically studied. To fill this gap, the authors introduce OSReward, a large, carefully curated, cross-platform benchmark of CUA trajectories with rigorously human-verified ground-truth verdicts. They also create OSReward-Hard, a challenge subset of genuinely difficult cases, and OSReward-Multi, which adds fine-grained scores for efficiency and alignment. Using this comprehensive benchmark, they evaluate 27 recent VLM judges and reveal that even state-of-the-art models have a strong leniency bias, often misclassifying failed runs that claim success as actual success. The only sufficiently reliable models are prohibitively expensive, while open models fall short in accuracy. To bridge this gap, the authors release OS-Shepherd-100K, a large open corpus of reasoning-annotated judgments, and train two OS-Shepherd reward models (9B and 35B parameters) that match commercial judges' accuracy but run at 30–60 times lower cost. Extensive analyses clarify failure modes, input dependencies, and robustness, setting the stage for more scalable, affordable, and standardized CUA reward modeling.

Key findings

• OSReward contains 1019 human-gold trajectories across web, mobile, Ubuntu, and Windows platforms, with 43% success and 57% failure labels.
• OSReward-Hard challenge subset (284 trajectories) boosts failure rate to 70% and specifically targets trajectories that even humans initially disagreed on.
• Latest frontier VLM judges (e.g., Claude-Opus-4-8, GPT-5.5) reach up to ~89.7% binary accuracy on OSReward but drop below 70% accuracy on OSReward-Hard.
• Judges share a dominant leniency bias, over-accepting incomplete but apparently successful trajectories, composing ≥48% of all errors per judge.
• Affordable open-weight judges lag by 15-20+ percentage points in accuracy compared to closed frontier judges.
• OS-Shepherd models (9B and 35B) trained on OS-Shepherd-100K match near frontier accuracy (85.6-86.1%) but run at 30-60× lower cost per 1,000 judgments.
• Failed trajectories tend to be longer (up to 100 steps) and more complex, making verification harder.
• Multi-annotator labeling and meta-review ensure high-quality gold verdicts and detailed failure taxonomy for analyzing judge errors.

Threat model

The study addresses the adversarial challenge of reliably verifying if a complex agent trajectory has truly accomplished a user-defined task across diverse platforms, under the assumption the adversary is the agent generating potentially incomplete or incorrect task completions combined with misleading narrative claims. The judge does not interact with the environment, cannot query external tools, and must rely solely on the recorded trajectory data including screenshots and agent reasoning. It cannot directly observe ground truth beyond the trajectory, nor can it be modified or corrupted during evaluation.

Methodology — deep read

1. Threat Model and Assumptions: The paper considers an adversarial scenario indirectly by assessing whether vision-language judging models can reliably distinguish success and failure in CUA trajectories without reliance on task-specific instrumentation or human annotation. The model must process the last N=5 states and corresponding agent thoughts/actions to produce a binary success/fail verdict and optionally multi-dimensional scores without access to the environment or tools. No direct manipulation or attack scenario on the judge itself is evaluated.

2. Data: The authors build a cross-platform data infrastructure spanning web (Chromium browser), Windows, Ubuntu, and Android mobile emulator environments. Each platform is initialized with realistic user states, files, application accounts, and distractor content to reflect realistic user scenarios. Around 1500 task instructions are authored and peer-reviewed to ensure grounding and answerability across all platforms. Tasks range in complexity and length (up to 100 steps). Four diverse agent families (Claude, Gemini, Kimi, Qwen) execute instructions multiple times, yielding real successes and failures. Raw trajectories capture screenshots, actions, and agent reasoning.

3. Annotation Pipeline: All surviving trajectories after automatic pre-filtering undergo triplicate human annotation applying a strict success criterion that requires environment-verified task completion, not mere agent claims. Where annotators disagree, meta-review by two senior reviewers deliberates final verdicts. Failures are labeled with a taxonomy covering reasoning, planning, perception, action, and memory errors. Successful trajectories are also rated for alignment to task intent and efficiency on 3-class scales. This pipeline yields 1019 gold-standard trajectories for evaluation.

4. OSReward Benchmarks: Multiple dataset views include the full set of 1019, OSReward-Hard (284 hard cases with elevated failure rate and annotation difficulty), and OSReward-Multi (440 successful trajectories with fine-grained scoring). Trajectories cover GUI-only and GUI+CLI action styles, numerous professional and routine tasks, and realistic cross-platform diversity.

5. Judging Protocol: Twenty-seven VLM judges are evaluated under a standardized protocol. Each judge receives the last 5 states, agent thoughts, and action texts, then outputs success/fail and optionally efficiency/alignment scores. The judges include closed frontier models (Claude-Opus-4-6/8, GPT-5.5, Gemini), open-weight large and small models (Kimi, Qwen variants), and efficient smaller ones. No task-specific tuning or step-level supervision is provided.

6. Metrics and Analysis: Accuracy is decomposed into success recall (detecting true successes), fail recall (detecting true failures), and balanced accuracy. Error analyses classify errors into over-accepts and over-rejects with sub-types using strong labelers plus human rechecks. Cost per 1000 judgments is computed to analyze cost-accuracy tradeoffs, important for scaling.

7. OS-Shepherd Training: An open dataset of 100K reasoning-annotated trajectory judgments, OS-Shepherd-100K, is curated combining public and internal data with labeling guided by judge study insights to focus on de-biasing dominant failure modes. Based on this, OS-Shepherd reward models with 9B and 35B parameters are trained with a two-stage recipe targeting better distinction of false successes. These models produce reward signals at scale, balancing accuracy and cost better than existing models.

Overall, the methodology systematically constructs a large, diverse, high-quality benchmark rooted in realistic environments and human-curated data, providing a rigorous, platform-agnostic evaluation of VLM judge reliability that isolates judge errors from agent and instruction artifacts. The pipeline from data collection, annotation, benchmark design, to model evaluation and training is comprehensive, reproducible, and grounded in detailed analyses.

Technical innovations

• OSReward: The first standardized, cross-platform benchmark with rigorously human-verified CUA trajectory labels spanning web, mobile, Ubuntu, and Windows.
• Identification and quantification of a universal leniency bias in VLM judges that causes them to over-accept failed tasks falsely claiming completion.
• Construction and release of OS-Shepherd-100K, a large open corpus of reasoning-annotated trajectory judgments designed to address failures revealed by OSReward.
• Training of OS-Shepherd reward models (9B and 35B) that approximate top commercial judge accuracy at 30–60× lower cost, enabling scalable CUA reward modeling.

Datasets

• OSReward — 1019 trajectories — human-verified, cross-platform private dataset
• OSReward-Hard — 284 trajectories — human-verified hard subset of OSReward
• OSReward-Multi — 440 trajectories — subset of OSReward with fine-grained efficiency and alignment annotations
• OS-Shepherd-100K — ~100,000 trajectory judgments — curated from >300K judge instances, partially public

Baselines vs proposed

• Claude-Opus-4-8 on OSReward: binary accuracy = 89.7% vs OS-Shepherd-35B = 85.6%
• GPT-5.5 on OSReward: binary accuracy = 89.5% vs OS-Shepherd-9B = 86.1%
• Small open-weight judges (e.g., Qwen3-VL-30B) on OSReward: binary accuracy = 69.4% vs OS-Shepherd-9B = 86.1%
• Frontier closed judges on OSReward-Hard: accuracy drops to ~67-70% vs OS-Shepherd models at ~57-64%
• Cost per 1000 trajectories: frontier judges >$100 vs OS-Shepherd models ~$1–3 with near frontier accuracy
• Fail recall (catching failures) of lenient judges as low as 46%, indicating prevalent over-acceptance of failures

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.28609.

Fig 3: From realistic environments to raw trajectories: annotators prepare the environments and

Fig 4: The annotation pipeline. Each pre-filtered trajectory is labeled by three independent annotators;

Fig 3 (page 5).

Fig 4 (page 5).

Fig 5 (page 5).

Fig 6 (page 5).

Fig 7 (page 5).

Fig 8 (page 5).

Limitations

• The judge evaluation focuses on accuracy of binary success/fail classification, with limited exploration of more nuanced reward signal properties for RL training dynamics.
• OSReward-Hard still only contains 284 trajectories, a smaller subset whose representativeness for all hard cases may be limited.
• The human annotation pipeline, while rigorous, is costly and may not scale further to ever-larger and evolving task domains without automation.
• No explicit robustness tests against adversarially crafted trajectories or out-of-distribution tasks are reported.
• The training of OS-Shepherd relies on heuristics guided by judge study findings rather than large-scale human reannotation, possibly limiting ultimate ceiling.
• While cross-platform, the data skew towards certain applications and environments (e.g., Windows, Ubuntu) may limit generalization to other increasingly popular platforms or agent styles.

Open questions / follow-ons

• How might judge models be improved to specifically counteract the pervasive leniency bias beyond current prompting and training strategies?
• What is the impact of judge accuracy and bias on downstream reinforcement learning performance and real-world agent behavior?
• Could integration of additional modalities or external environment affordances improve judge reliability, especially on hard/ambiguous cases?
• How well do OS-Shepherd models generalize to novel tasks, domains, or agents not covered in the OSReward data?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this work offers critical insights into the reliability limits of current vision-language reward models when judging complex multi-step task completions, a scenario analogous to assessing user authenticity and interaction intent. The discovery of systematic leniency bias suggests automated scoring systems can be vulnerable to false acceptance if overly trusting user claims, underscoring the importance of rigorous evaluation benchmarks like OSReward when developing interpretable, trustworthy reward signals. Furthermore, the introduced low-cost OS-Shepherd models provide practical, scalable alternatives to costly commercial judges, enabling more widespread deployment of reliable reward models for evaluation and training in continuous interaction contexts. Overall, integrating such benchmarked reward models with multi-modal interaction data in bot-defense pipelines may improve the discrimination between genuine user task success and scripted or automated failure attempts masked by plausible narrative.

Cite

@article{arxiv2607_28609,
  title={ OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models },
  author={ Qiushi Sun and Kanzhi Cheng and Yian Wang and Bowen Yang and Hang Yan and Liheng Chen and Fangzhi Xu and Zichen Ding and Nuo Chen and Jialin Cao and Xingdong Gong and Zehao Li and Kaiming Jin and Xinfeng Yuan and Zhoumianze Liu and Jingyang Gong and Zhangyue Yin and Jiahui Gao and Zhiyong Wu and Tianbao Xie and Jianbing Zhang and Ben Kao and Lingpeng Kong },
  journal={arXiv preprint arXiv:2607.28609},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.28609}
}

Read the full paper

• arXiv:2607.28609 (abstract)
• arXiv:2607.28609 (PDF)