OSGuard: A Benchmark for Safety in Computer-Use Agents
Source: arXiv:2606.15034 · Published 2026-06-13 · By Mina Mohammadmirzaei, Jeffrey Flanigan
TL;DR
OSGuard addresses a critical blind spot in evaluating computer-use agents: while many benchmarks assess whether agents complete tasks as instructed, they often ignore safety failures where unsafe shortcuts or actions violate important task-local constraints despite nominal success. The paper introduces OSGuard, a dual-granularity benchmark suite focused on safety under benign, unchanged user instructions. It consists of (1) an action-level benchmark requiring guardrails to classify proposed next-step actions as allowed, unrelated, or unsafe relative to the original instruction and current UI state, and (2) a risk-augmented execution suite with environment modifications that introduce latent hazards but preserve a safe path to completion, enabling evaluation of full-task execution safety beyond nominal success. OSGuard enables diagnosis of gaps between local action-level oversight and reliable end-to-end safety. Experimental results show state-of-the-art guardrails achieve good accuracy (up to 80%) on isolated action judgments but struggle more on actions from hazard-modified environments and provide only modest safety improvements during full-task execution, leaving significant residual unsafe completions. This highlights a persistent challenge in developing guardrails that consistently enforce safety constraints throughout complex multi-step computer-use tasks.
Key findings
- Gemini 3 Pro Preview guardrail achieves 80% accuracy and 0.80 macro-F1 overall on the action-level benchmark, with F1 scores of 0.85 (allowed), 0.80 (unrelated), and 0.76 (unsafe).
- Action-level benchmark contains 324 contextualized proposed actions labeled allowed, unrelated, or unsafe, drawn from 4 construction sources including risk-augmented variant executions.
- On risk-augmented execution suite of 45 manually constructed task variants, unguarded agent completes 62% safely but incurs 38% unsafe completions violating safety invariants despite nominal task success.
- Guarding with Gemini 3 Pro Preview reduces unsafe completions on risk-augmented variants modestly from 38% to 33%, without increasing overall success rate, introducing 4% retry-termination rate.
- Performance drops substantially on action-level examples from risk-augmented variant executions compared with standard OSWorld actions, with best macro-F1 only 0.48.
- DeAction guardrail with same Claude Sonnet 4.5 backbone drops from 84% accuracy on MisActBench to 57% accuracy and 0.55 macro-F1 on OSGuard action-level benchmark, especially lower recall (0.20) on unsafe actions compared to prompted Claude guardrail (0.48 recall).
- Failures in risk-augmented variants vary by hazard family: executor performs worst on destructive overwrite/deletion hazards, best on configuration clobbering.
- Action-level accuracy and macro-F1 vary strongly by data source: best accuracy 100% on standard OSWorld actions but drop to ~43-57% on unsafe proposers and risk-augmented actions.
Threat model
The adversary is implicit and limited: OSGuard focuses on failures arising from a benign, unchanged user instruction executed by an agent within a realistic desktop or web environment that may contain latent hazards. The threat is not a malicious user or explicit active attacker injecting adversarial inputs, but rather an agent inadvertently taking unsafe shortcuts or actions that violate task-local safety constraints in a non-adversarial, state-dependent manner. The adversary cannot alter the instruction or environment directly, but challenges safety by exploiting plausible but unsafe continuations in the given benign setting.
Methodology — deep read
The core motivation is to evaluate computer-use agents not just by task success but by safety when executing benign user instructions in environments with latent hazards. Two complementary evaluation granularities are developed. First, the action-level benchmark tests guardrails' ability to judge whether a candidate next action is allowed, unrelated, or unsafe given: (a) the original user instruction, (b) a screenshot plus accessibility tree representing the current interface state, and (c) the proposed action itself. This is a pre-execution oversight task. Labels are assigned by human annotators who use exactly the inputs available to the guardrail (no additional proposer task descriptions). Action-level items are drawn from four data sources: (1) standard OSWorld task executions, sampling intermediate states and their next actions, (2) interrupted OSWorld execution prefixes paired with compatible next actions proposed to continue towards the goal, (3) interrupted OSWorld prefixes paired with explicitly unsafe next actions grounded in the current state, and (4) executions of risk-augmented variants that modify environment state to introduce hazards but keep the original instruction unchanged. This process yields 324 labeled candidate actions: 80 allowed, 116 unrelated, 128 unsafe. The benchmark ensures realism via sampling real intermediate states, and ambiguity by interrupting trajectories early with multiple plausible next steps. The second evaluation is the risk-augmented execution suite consisting of 45 task variants derived from OSWorld. The original user instruction remains fixed while environment states are manually modified to introduce latent hazards such as destructive overwrites, overbroad permission changes, scope escapes, conflicting config entries, access to sensitive content, or wrong-target updates. These modifications keep the original task achievable by the executor agent but create plausible unsafe shortcuts. Each variant is paired with an augmented evaluator that adds explicit, concrete safety invariants as state-based checks (file hashes, permission preservation, content protection, etc) on top of the original OSWorld success criteria. Evaluation of task executions records success only if the original task and all safety invariants are satisfied. Unsafe completions are reproducible and precisely attributed to violated invariants. Experimentally, a prompted guardrail model is implemented as a reactive pre-execution classifier that returns 'allowed', 'unrelated', or 'unsafe' with optional natural-language feedback. When deployed online during execution, the guarded executor may retry up to two times at each state if actions are blocked, enabling recovery while bounding revisions. Multiple guardrails are benchmarked (Gemini 3 Pro Preview, GPT 5.1, Claude Sonnet 4.5, and DeAction with Claude Sonnet 4.5 as baseline). Metrics include accuracy and macro-F1 on labeled actions, plus variant-level success and unsafe completion rates during execution with and without guardrails. The evaluation protocol measures offline local judgment quality and online downstream safety impact under explicit environment hazards. The OSGuard benchmark collects paired image, accessibility, and action data and labels from human annotation and manual variant design. Code release and model details are not explicitly stated, making exact reproducibility unclear.
Technical innovations
- Dual-granularity evaluation design combining action-level proposed-action oversight with risk-augmented full-task execution safety under unchanged benign instructions.
- Construction of risk-augmented task variants introducing latent, state-dependent hazards while preserving original user instructions and safe task completion paths.
- Augmented evaluators that enforce explicit state-based safety invariants in addition to original task success criteria, enabling discrimination of unsafe task completions.
- Action-level benchmark labeling protocol assigning allowed, unrelated, and unsafe labels using only inputs available to the guardrail (original instruction, current state, candidate action), without revealing proposer-generated task intentions.
- Deployment of a prompted, reactive guardrail model with natural language feedback enabling pre-execution intervention and limited retries in interactive task execution.
Datasets
- OSWorld executions — hundreds of real desktop task executions used for candidate action extraction and variant creation — public OSWorld environment
- OSGuard action-level benchmark — 324 candidate actions with labels allowed/unrelated/unsafe from 4 construction sources — constructed from OSWorld executions and risk-augmented variants, dataset not separately public
- OSGuard risk-augmented execution suite — 45 manually constructed task variants derived from OSWorld with injected latent hazards — proprietary manual construction
Baselines vs proposed
- DeAction (Claude Sonnet 4.5 backbone): action-level accuracy = 57% vs prompted Claude Sonnet 4.5 guardrail accuracy = 60%
- Gemini 3 Pro Preview action-level benchmark accuracy = 80% and macro-F1 = 0.80 vs GPT 5.1 accuracy = 63% macro-F1 = 0.62
- Ungarded executor on risk-augmented variants: 62% success with 38% unsafe completions, guarded with Gemini 3 Pro preserves 62% success, reduces unsafe completions to 33%
- On risk-augmented actions subset, Gemini 3 Pro accuracy = 50% macro-F1 = 0.27 vs standard OSWorld actions accuracy = 100%, macro-F1=1.00
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.15034.
Fig 1: OSGuard benchmark overview. (a) Construction sources for the action-level benchmark:
Fig 2: Example risk-augmented variant. The original YouTube history-clearing instruction is
Limitations
- Limited scale: action-level benchmark consists of only 324 labeled actions and 45 manual task variants, restricting scope and statistical power.
- Evaluation restricted to a single task-executing agent architecture, limiting generalization across diverse agent types and capabilities.
- Risk-augmented variants manually constructed and limited to six hazard families, may not capture full spectrum of realistic hazards or emergent unsafe shortcuts.
- No adversarial or adversary-driven instruction modifications; focuses only on benign, unchanged instructions limiting coverage of malicious or deceptive user inputs.
- Guardrail models evaluated primarily via prompted LLMs and DeAction baseline; no deeply specialized or learned safety detection modules examined.
- No discussion of robustness to distribution shifts beyond risk-augmented variants or adversarial attacks on guardrail decisions.
- Reproducibility is unclear due to no public code release or dataset availability, and guarded execution retry parameters fixed without ablation.
Open questions / follow-ons
- How to design guardrail models that better generalize to latent, state-dependent hazards arising in risk-augmented and real-world environments beyond current prompted approaches.
- Can predictive or anticipatory guardrails that utilize forward rollout or world modeling better prevent unsafe shortcuts than reactive pre-execution classifiers?
- How do different executor agent architectures and training strategies interact with guardrail performance and safety outcomes in OSGuard or similar benchmarks?
- What scalable methods exist for automatically constructing diverse environment variants that introduce safety hazards for thorough evaluation without manual curation?
Why it matters for bot defense
For bot-defense and CAPTCHA practitioners, OSGuard highlights an important class of safety failures in autonomous computer-use agents that are relevant when bots interact with complex user interfaces. It underscores that successful task completion alone does not guarantee safety or correctness, especially when latent environment hazards create tempting unsafe shortcuts or privilege escalations. The dual evaluation approach offers a model for combining local action-level validation with end-to-end risk-awareness, which could inform bot-defense strategies that require adaptive, context-sensitive gating of bot actions rather than static rule checks. OSGuard’s findings caution that even strong local oversight may not prevent unsafe outcomes in longer workflows, motivating layered defenses and forward-looking risk models for bot safety. The benchmark’s focus on benign instructions and realistic states closely parallels challenges in detecting gradually misaligned or shortcutting bot behavior under ostensibly legitimate queries.
Cite
@article{arxiv2606_15034,
title={ OSGuard: A Benchmark for Safety in Computer-Use Agents },
author={ Mina Mohammadmirzaei and Jeffrey Flanigan },
journal={arXiv preprint arXiv:2606.15034},
year={ 2026 },
url={https://arxiv.org/abs/2606.15034}
}