Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents

Source: arXiv:2607.28227 · Published 2026-07-30 · By Hanzhang Zhou, Panrong Tong, Xu Zhang, Quyu Kong, Chenglin Cai, Tianyu Xia et al.

TL;DR

Qwen-UI-Agent addresses major limitations of current GUI agents by targeting real-world deployment scenarios with complex, cross-platform digital task execution. Unlike previous agents focused mainly on simulated environments, Qwen-UI-Agent operates on a large-scale real-device mobile runtime with over 100 physical devices and 150+ apps, combined with scaled sandbox environments for mobile, desktop, browser, and DeepSearch tasks. It unifies GUI, CLI, and API action spaces, enabling efficient batched executions and cross-domain workflows that span mobile, computer, web, and search platforms. A key innovation is an AutoResearch-style agent-driven data flywheel that automates task synthesis, failure diagnosis, and iterative capability improvements using online reinforcement learning at unprecedented scale (10,000 concurrent environments) for long-horizon trajectories exceeding 100 interaction steps.

Empirically, Qwen-UI-Agent sets new state-of-the-art success rates on multiple real-device mobile benchmarks including 92.2% on MobileWorld-Real and 97.5% on AndroidDaily, outperforming leading closed-source models such as Gemni 3.1 Pro, GPT-5.6 Sol, and Claude Opus 4.8 by several percentage points. It also delivers strong results on computer-use and browser GUI benchmarks while supporting advanced agentic capabilities like proactive service initiation and complex cross-device workflows. This work represents a significant leap toward practical foundation GUI agents that reliably execute real-world tasks requiring diverse modes of interaction, multi-step reasoning, and continuous autonomous improvement.

Key findings

• On MobileWorld-Real (400+ tasks on 100+ apps, real devices), Qwen-UI-Agent achieves 92.2% success, surpassing Gemini 3.1 Pro (86.2%), Claude Opus 4.8 (84.7%), GPT-5.6 Sol (85.4%), and Seed 2.1 Pro (88.7%).
• On AndroidDaily real-device benchmark, the agent achieves near-perfect 97.5% success rate.
• On MobileWorld simulated benchmark, Qwen-UI-Agent scores 82.1%, outperforming Opus 4.8 by 14.6 points.
• On OSWorld-Verified computer-use benchmark, Qwen-UI-Agent attains 79.5%, ranking second overall and beating GPT-5.5 and Gemini 3.1 Pro.
• On OSWorld-v2, Qwen-UI-Agent achieves 13.9% binary success and 40.0% partial-progress, exceeding MiniMax M3 and Qwen 3.7 Plus by over 17 points in partial-progress while requiring 21.7%–58.4% fewer steps per task.
• On WebArena browser benchmark, it achieves 73.6%, outperforming Claude Opus 4.8 (71.9%), GPT-5.5 (69.5%) and Gemini 3.1 Pro (65.3%).
• Over 40% of action outputs in computer-use are batched GUI+CLI commands, substantially shortening trajectories and improving efficiency.
• The real-device mobile runtime health-aware scheduler manages over 100 physical devices with virtual displays, blacklisting unhealthy devices to maintain stable training and evaluation environments.

Threat model

The system assumes an adversary limited to causing environmental faults such as network failures, unexpected pop-ups, or UI changes that do not explicitly control the agent or underlying device beyond normal usage constraints. The agent cannot bypass user approval mechanisms or directly manipulate hardware outside GUI/CLI commands. The threat model focuses on robustness to real-world operational challenges rather than adversarial attacks or malicious interference.

Methodology — deep read

Qwen-UI-Agent is designed as a unified foundation agent to execute interactive digital tasks spanning mobile devices, desktop computers, browsers, and DeepSearch systems.

Threat Model & Assumptions: The adversary model implicitly assumes a challenging real-world setting where the agent must cope with unpredictable UI layouts, network instability, permission pop-ups, and execution interruptions. It operates with no direct API access beyond defined api_call actions and cannot arbitrarily control hardware beyond GUI and CLI commands. User confirmation and manual takeover are enforced for sensitive operations.

Data: The training data derives from a combination of large-scale sandbox trajectory generation and real-device execution logs. The real-device mobile runtime includes over 100 physical devices with 150+ apps, capturing complex real-world states and failures. Sandboxed simulators span mobile (MobileWorld environment rebuilt using redroid containers), computer (Ubuntu VM OSWorld with GUI+CLI), browser (Playwright Chromium), and DeepSearch retrieval tasks. Task datasets include MobileWorld, MobileWorld-Real, AndroidDaily, OSWorld-Verified, OSWorld-v2, WebArena, DeepSearch, and GUI grounding benchmarks like ScreenSpot-Pro.

Architecture/Algorithm: The agent uses a single unified policy π_θ that takes as input the user instruction, multichannel observations (screenshots, CLI output, API responses), and prior history. The action space merges GUI actions (click, type, drag, open app, system button), CLI commands, API calls, and user interaction commands (ask_user, terminate). It supports producing either single or batched sequences of actions per model step to improve efficiency. The agent applies a vision-language reasoning pipeline to jointly interpret GUI states and CLI/API information.

Training Regime: Training combines supervised fine-tuning on labeled trajectories with online reinforcement learning guided by a verifier module. RL trains on long-horizon episodes with over 100 decision steps, leveraging around 10,000 parallel sandbox environments to accelerate rollout generation. A curriculum learning strategy prioritizes intermediate difficulty tasks and replaces mastered tasks gradually. Human involvement is minimized via an agent-driven AutoResearch data flywheel that autonomously generates and validates new tasks and environments.

Evaluation Protocol: Evaluation covers both real-device benchmarks and large-scale sandbox suites for mobile, computer, and web use, measuring success rates and partial-progress metrics. Real-device evaluation uses a health-aware scheduler to assign tasks to healthy phones and automatically blacklists failing devices. An AI-based AutoJudge inspects full trajectories to separate model failures from environment failures. Baselines include cutting-edge closed-source models (Gemini 3.1 Pro, Claude Opus 4.8, GPT-5.6 Sol) and open models (Seed 2.1 Pro, MiniMax M3). Ablations evaluate the impact of batched actions, hybrid GUI+CLI spaces, and online RL.

Reproducibility: Code and models are not publicly released. The paper provides extensive descriptions of environment setups and evaluation benchmarks but does not release frozen weights. Key datasets including MobileWorld-Real and AndroidDaily are introduced but sources are not explicitly public.

Example Walkthrough: For a flight cancellation scenario, the agent perceives a mobile notification, queries alternate flights via API, obtains user approval for rebooking via mobile GUI, updates related schedules on desktop through GUI and CLI actions, and ultimately sends updated information. This exemplifies cross-platform workflows, multimodal observations, batched hybrid action execution, and proactive task initiation via the harness layer.

Overall, the methodology advances GUI agents beyond simulation by tightly integrating real-device deployment, multi-modal action spaces, scalable online RL over long horizons, and an iterative automated data generation and capability improvement pipeline.

Technical innovations

• Development of a large-scale real-device mobile runtime encompassing 100+ physical devices with virtual displays and health-aware scheduling to bridge the sim-to-real gap.
• Unified hybrid action space interleaving GUI interactions with CLI commands and API calls, supporting batched multi-action decisions within a single model step.
• Agent-driven AutoResearch-style data flywheel that automatically constructs tasks, diagnoses failures, and iteratively refines agent capabilities with minimal human intervention.
• Scaling of verifier-guided online reinforcement learning to training on trajectories exceeding 100 interaction steps across 10,000 parallel environments for long-horizon task mastery.
• Lightweight harness layer enabling proactive service initiation from passive notifications and seamless cross-platform, multi-device workflow execution.

Datasets

• MobileWorld — simulated Android sandbox with 20 apps — proprietary Alibaba
• MobileWorld-Real — 400+ real-device tasks on 100+ physical mobile devices and 150+ apps — constructed by authors
• AndroidDaily — real-device mobile usage benchmark — constructed by authors
• OSWorld-Verified — Ubuntu VM computer use benchmark — OSWorld project
• OSWorld-v2 — extended OSWorld computer benchmark with partial-progress metrics — OSWorld project
• WebArena — browser GUI benchmark with DOM-based verifiers — proprietary
• DeepSearch — multi-source web search and retrieval tasks — integrated third-party APIs
• ScreenSpot-Pro — GUI grounding benchmark for visual interface comprehension — proprietary

Baselines vs proposed

• Gemini 3.1 Pro: 86.2% success on MobileWorld-Real vs Qwen-UI-Agent: 92.2%
• Claude Opus 4.8: 84.7% success on MobileWorld-Real vs Qwen-UI-Agent: 92.2%
• GPT-5.6 Sol: 85.4% success on MobileWorld-Real vs Qwen-UI-Agent: 92.2%
• Seed 2.1 Pro: 88.7% success on MobileWorld-Real vs Qwen-UI-Agent: 92.2%
• Opus 4.8: 67.5% on MobileWorld vs Qwen-UI-Agent: 82.1%
• GPT-5.5: below 79.5% on OSWorld-Verified vs Qwen-UI-Agent: 79.5%
• MiniMax M3 partial-progress OSWorld-v2: 22.3% vs Qwen-UI-Agent: 40.0%
• Qwen 3.7 Plus partial-progress OSWorld-v2: 21.5% vs Qwen-UI-Agent: 40.0%
• Claude Opus 4.8: 71.9% on WebArena vs Qwen-UI-Agent: 73.6%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.28227.

Fig 1: Qwen-UI-Agent demonstrates leading or competitive performance across diverse GUI settings.

Fig 2 (page 1).

Fig 2: An illustrative trajectory of Qwen-UI-Agent for proactive cross-platform task execution. The left

Fig 4 (page 5).

Fig 5 (page 5).

Fig 6 (page 5).

Fig 7 (page 5).

Fig 8 (page 5).

Limitations

• No public release of training code, models, or datasets limits reproducibility and external validation.
• Evaluation focuses on success rates measured via verifiers and AutoJudge AI, which may not fully capture nuanced user experience or failure modes.
• Adversarial robustness assessment or evaluation under active malicious manipulation is not reported.
• Real-device runtime is limited to Chinese mobile ecosystem and select apps, which may not generalize to other regional app markets or platforms.
• Some environment failures in real-device training require manual maintenance and blacklisting, indicating system fragility.
• Impact of large-scale online RL on model stability and catastrophic forgetting remains unclear.

Open questions / follow-ons

• How to generalize real-device runtime and training to diverse global app ecosystems beyond the Chinese market?
• What new verification methods can better detect subtle task failures or degraded UX beyond binary success metrics?
• How can agents actively recover from adversarial UI manipulations or spoofed digital signals?
• What is the upper bound on task length and complexity accessible via scaled online RL before model stability or sample efficiency issues emerge?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, Qwen-UI-Agent highlights the increasing sophistication of foundation models that can interact across GUI, CLI, and API modalities to autonomously execute complex, long-horizon workflows spanning multiple devices and platforms. This suggests attackers or automated agents could leverage similar architectures to bypass conventional CAPTCHA gates via hybrid strategies combining visual GUI manipulation with programmatic CLI commands.

Defenders should note the importance of real-device evaluation and health-aware environment monitoring to detect model failures versus environment manipulation or instability. The agent’s ability to proactively initiate services based on ambient device signals highlights the potential for evasive automated behaviors triggered by system notifications or changes. Understanding the interplay of batched actions, multi-modal perception, and continuous learning reinforces the need for multifactor anomaly detection approaches that span both user interaction patterns and command execution traces. The paper provides a foundation for assessing next-generation automated agents’ capabilities and points toward integrated defense techniques targeting hybrid GUI+CLI workflows.

Cite

@article{arxiv2607_28227,
  title={ Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents },
  author={ Hanzhang Zhou and Panrong Tong and Xu Zhang and Quyu Kong and Chenglin Cai and Tianyu Xia and Gongjie Zhang and Jianan Zhang and Long Li and Long Chen and Lei Wang and Gaole Dai and Pengxiang Li and Liangyu Chen and Yue Wang and Steven Hoi },
  journal={arXiv preprint arXiv:2607.28227},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.28227}
}

Read the full paper

• arXiv:2607.28227 (abstract)
• arXiv:2607.28227 (PDF)