Qwen-CUA: Native Computer Use for (almost) Everything

Source: arXiv:2608.02352 · Published 2026-08-03 · By Dunjie Lu, Shuai Bai, Tianyi Bai, Sicheng Fan, Chang Gao, Jian Guan et al.

TL;DR

Qwen-CUA addresses the challenge of creating large-scale AI agents that can operate native desktop and web software solely through visual input (screenshots) and low-level keyboard and mouse events, without reliance on structured metadata, DOM trees, accessibility APIs, or application-specific toolkits. This approach enables an AI to use almost any software exactly as a human would, but poses difficult problems of long-horizon state tracking, sparse verifiable rewards, and diverse environment coverage. Qwen-CUA uses a 397-billion-parameter Qwen mixture-of-experts large language model backbone and a scalable cloud rollout infrastructure with nearly 100,000 vCPUs to generate tens of thousands of verifiable interaction tasks across broad software domains, including personalized, professional, and scientific applications.

The system manages long visual histories by maintaining a sliding window of 20 active screenshots, folding older screenshots into stable text placeholders to preserve recent visual context and improve inference cache reuse. It jointly optimizes supervised and reinforcement learning with Soft Adaptive Policy Optimization, using fully verifiable terminal rewards without intermediate shaping. Iterative training cycles refresh data and calibrate task difficulty to push learning frontiers. Experiments across eight benchmarks demonstrate substantial gains over the prior Qwen-3.7 model, with OSWorld-Verified task success reaching 86.2% (up from 73.3%). A 1-trillion-parameter scale-up named Qwen-CUA-Max further improves results. The agent also achieves improved robustness to prompt-injection attacks measured by RedTeamCUA.

Overall, the work demonstrates a scalable approach to building a general native computer-use AI agent that can operate arbitrary software through visual-haptic interaction alone, sets new state of the art on diverse computer-use benchmarks, and highlights the importance of scalable verifiable tasks, long-horizon trajectory slicing, and hybrid tool integration for practical deployment.

Key findings

• Qwen-CUA achieves 86.2% task success on the OSWorld-Verified benchmark, outperforming Qwen3.7 (73.3%), GPT-5.5 (78.7%), and Claude Opus 4.8 (83.4%).
• On OSWorld 2.0, Qwen-CUA obtains 18.5% binary completion and 48.4% partial completion, a large improvement over Qwen3.7 at 2.5% / 22.5%.
• Scaling to a 1-trillion-parameter Qwen-CUA-Max further improves OSWorld-Verified to 87.6% and OSWorld 2.0 to 21.2% / 53.3%.
• Qwen-CUA reduces RedTeamCUA indirect prompt injection attack success rate from 36.6% (Qwen3.7) to 16.4%, improving safety while boosting benign task completion (74.0% vs 70.5%).
• Qwen-CUA maintains an active visual history of 20 screenshots and folds older screenshots in blocks of 10, enabling long-horizon state tracking and stable prompt prefixes that improve inference cache reuse (KV-cache).
• Soft Adaptive Policy Optimization (SAPO) enables stable reinforcement learning on long multimodal trajectories by smoothing importance sampling ratios and asymmetrically penalizing off-policy negative-advantage updates.
• Iterative training with refreshed supervised fine-tuning data and calibrated RL tasks realizes continual improvement, demonstrated by steady score increases across development checkpoints (Fig 4b).
• Hybrid interaction combining native computer use with Bash shell commands shortens average trajectory turns on MyPCBench tasks from 63.6 to 49.1 for Qwen-CUA, trading some task performance for efficiency.

Threat model

The adversary attempts indirect prompt injection attacks via on-screen textual content in composite web–operating-system hybrid tasks, aiming to manipulate the agent's behavior. The attacker lacks internal access to the model’s parameters or privileged OS privileges and can only manipulate the environment state visible through screenshots. The threat model assumes the agent cannot be tricked by crafted visual inputs outside native screenshots and has no external connectivity vulnerabilities. It does not address physical attacks or direct software exploits on the agent infrastructure.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is modeled primarily in RedTeamCUA, which exposes the agent to indirect prompt-injection attacks via on-screen content. The attacker cannot directly access internal model state or training data and actions must be mediated through keyboard/mouse interaction on screenshots only. The agent assumes no access to DOM, accessibility metadata, or privileged APIs, restricting its input-output bandwidth to realistic human-computer visual-haptic control.

2. Data & Environment: The training uses approximately 40,000 verifiable tasks spanning synthetic mock web services, diverse sandboxed professional desktop applications, and personalized workflows collected from human annotators' real desktops. Tasks include environment operation, user-interactive clarification dialogues (simulated users), and long-horizon multi-phase workflows with verifiable phase completion. Data is split by task type and domain, with environment snapshots resettable for independent rollouts. Human trajectories include detailed step-level action annotations augmented with model-generated reasoning rationale.

3. Architecture / Algorithm: Qwen-CUA uses a large-scale Qwen mixture-of-experts (MoE) language model backbone of 397 billion parameters with 17 billion active experts. The model receives as input a sequence consisting of task instructions, up to 20 active screenshots, and folded textual placeholders for older screenshots. Outputs are keyboard and mouse actions drawn from a native interface action space. The system manages context length by chunked folding: every 10 steps the oldest 10 screenshots are replaced with text placeholders to preserve reasoning and improve KV-cache reuse.

Reinforcement learning uses Soft Adaptive Policy Optimization (SAPO) which smooths importance sampling weight clipping with a temperature-controlled gate, asymmetrically penalizing policy updates from negative-advantage off-policy tokens more than positive ones. The RL loss applies only to model-generated tokens, excluding context from loss gradients. Trajectory slicing converts a full episode to overlapping context-bounded slices, each inheriting full episode reward to allow credit assignment to steps despite sparse terminal rewards.

4. Training Regime: Training proceeds in iterative cycles alternating supervised fine tuning on mixtures of human and model-generated data with reinforcement learning on calibrated verifiable tasks. SFT data is refreshed by re-running teacher policy rollouts on unresolved queries and weak domains, including new human trajectories. RL task distributions are recalibrated by filtering out trivially solvable or unreachable tasks based on rollout success rates. The distributed training leverages a massive Alibaba Cloud ECS rollout fleet with ~100,000 vCPUs running tens of thousands of concurrent browser and desktop sandbox environments.

5. Evaluation Protocol: The agent is evaluated under a pure native computer-use protocol restricting input to screenshots and output to keyboard/mouse actions. Eight benchmarks were used spanning everyday desktop workflows (OSWorld-Verified, OSWorld 2.0), personalized workflows (MyPCBench), professional/scientific software (ScienceBoard), macOS-specific tasks (MacAgentBench), broad multi-application domains (Gym-Anything), real web interactions (WebArena), and adversarial attack robustness (RedTeamCUA). Metrics include task success rates, strict binary completion, partial completion, attack success rate, and efficiency measures (output tokens and agent turns per task). Efficiency analyses compare output verbosity and interaction length.

6. Reproducibility: Authors released code at https://github.com/xlang-ai/Qwen-CUA. The environment setup including webs mocks and sandboxed professional application installations is proprietary. Model weights and exact checkpoints are not stated as public but may be released by the authors. Evaluation scripts and benchmark integration details are provided in appending documents. The large-scale infrastructure and task sets pose significant replication challenges for most groups.

End-to-End Example: Given an OSWorld task, Qwen-CUA receives an initial desktop screenshot and task specification. It visually parses the interface, then emits a sequence of mouse clicks and keystrokes to navigate menus, input data, and change window focus, while maintaining context of prior screenshots and folded older states. After about 200 agent turns across 20 active screenshots, it reaches the final state which an executable evaluator verifies as task success, providing a binary reward used to update the policy by SAPO via trajectory slicing. This cycle repeats with new tasks and human-annotated workflows augmenting data, iteratively improving the model's performance across domains.

Technical innovations

• A native computer-use agent architecture receiving only screenshots and outputting keyboard-mouse actions without any DOM, accessibility metadata, or task-specific APIs.
• Long-horizon visual context management via an active window of 20 screenshots plus blockwise chunked folding of older history into stable textual placeholders for KV-cache efficiency.
• Soft Adaptive Policy Optimization (SAPO) that smooths importance ratio clipping asymmetrically to stabilize reinforcement learning on long multimodal interaction trajectories.
• Iterative training loop that uses the current policy to identify unresolved queries and weak domains to refresh the supervised fine-tuning data and recalibrate RL task distributions for continual improvement.
• Large-scale verifiable task design using executable evaluators and phase-state chaining to construct complex multi-phase workflows with reliable outcome feedback.

Datasets

• About 40,000 verifiable computer-use interaction tasks from diverse environments — proprietary Alibaba Cloud ECS rollout infrastructure
• Personalized long-horizon workflows from human annotators spanning everyday and professional desktop applications — proprietary
• Synthetic mock web service environments implementing product interfaces with programmatic state control — proprietary

Baselines vs proposed

• Qwen-3.7: OSWorld-Verified score = 73.3% vs Qwen-CUA: 86.2%
• GPT-5.5: OSWorld-Verified score = 78.7% vs Qwen-CUA: 86.2%
• Claude Opus 4.8: OSWorld-Verified score = 83.4% vs Qwen-CUA: 86.2%
• Qwen-3.7: OSWorld 2.0 binary / partial completion = 2.5% / 22.5% vs Qwen-CUA 18.5% / 48.4%
• Qwen3.7: RedTeamCUA benign task success / attack success = 70.5% / 36.6% vs Qwen-CUA 74.0% / 16.4%
• Qwen-CUA-Max (1T+ params) improves OSWorld-Verified to 87.6% and OSWorld 2.0 to 21.2% / 53.3%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2608.02352.

Fig 1: Main results across eight computer-use benchmarks. OSWorld 2.0 reports binary completion (dark) and

Fig 2: Native computer-use

Fig 3: Long-horizon context management. (a) Active visual history scales to 20 screenshots. (b) Chunked

Fig 6: Agentic efficiency along two dimensions: (a) token efficiency on OSWorld-Verified, measured by

Fig 10: follows a single real-world trajectory from the internal Chrome extension. Given a natural-

Fig 6 (page 23).

Fig 7 (page 23).

Fig 8 (page 23).

Limitations

• The system relies entirely on screenshots and native input, which impedes access to semantic metadata and internal app state that could improve efficiency.
• Evaluation benchmarks do not fully capture real-world distribution shifts such as unfamiliar GUI layouts or highly dynamic content; deployment shows edge failures.
• Task generation and environment setups are largely proprietary and closed-source, restricting immediate community reproducibility.
• RedTeamCUA results show reduced but nonzero attack success indicating remaining vulnerabilities to indirect prompt injections.
• Interaction efficiency metrics are conflated by differences in action batching between proprietary baselines and single-action Qwen-CUA outputs.
• Scaling beyond one trillion parameters and cloud infrastructure requirements limit accessibility for most research groups.

Open questions / follow-ons

• How can semantic metadata (e.g., accessibility trees) be safely integrated to augment native computer use while preserving broad application coverage?
• What are improved methods for automated summarization or compression of long visual interaction histories beyond fixed chunk folding?
• How can indirect prompt-injection robustness be further enhanced or formally verified in visually grounded interactive agents?
• To what degree can hybrid tool use combining CLI, API, and native visual control be optimized jointly to maximize both efficiency and capability?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, Qwen-CUA demonstrates that large-scale AI agents can use native visual interfaces to operate arbitrary software environments with minimal privileged information, revealing new attack surfaces and challenges. The visual-only modality, long horizon context tracking, and verifiable reward design highlight the complexity of building robust, adaptable agents that resist environment manipulations including indirect prompt injections. Incorporating extensive rollout infrastructure with massive concurrent environments illustrates the engineering scale required to build trustworthy native computer-use AI.

Practitioners should note the demonstrated improvements in attack success rate reduction and task completion under adversarial conditions, signaling the value of specialized robust training and evaluation benchmarks like RedTeamCUA. The chunked folding approach to visual history management and the SAPO reinforcement learning algorithm may inspire new defenses that monitor state consistency or verify user intent across extended interactions in CAPTCHA or bot-mitigation settings. However, residual vulnerabilities and interaction inefficiencies suggest ongoing need for monitoring, layered defenses, and controlled tool or command line integration to reduce risk in real deployments.

Cite

@article{arxiv2608_02352,
  title={ Qwen-CUA: Native Computer Use for (almost) Everything },
  author={ Dunjie Lu and Shuai Bai and Tianyi Bai and Sicheng Fan and Chang Gao and Jian Guan and Feng Hu and Mianqiu Huang and Xingyang Huang and Yizhen Jiang and Yuheng Jing and Dehui Kong and Ning Li and Dayiheng Liu and Shixuan Liu and Zheng Liu and Que Shen and Bowen Wang and Junli Wang and Chencan Wu and Rui Xie and Tianbao Xie and Zhihui Xie and Haiyang Xu and An Yang and Tao Yu and Wenzhen Yuan and Xi Zhang and Zhenru Zhang and Mingkang Zhu and Zhaoqing Zhu and Yizhong Cao and Kai Dang and Binyuan Hui and Kaixin Li and Junyang Lin and Haiquan Wang and Zekun Wang and Yiheng Xu and Fan Yan and Mengqi Yuan and Danyang Zhang and Jiajun Zhang and Zhipeng Zhang and Fan Zhou and Fan Zhou },
  journal={arXiv preprint arXiv:2608.02352},
  year={ 2026 },
  url={https://arxiv.org/abs/2608.02352}
}

Read the full paper

• arXiv:2608.02352 (abstract)
• arXiv:2608.02352 (PDF)