Xiaomi-GUI-0 Technical Report

Source: arXiv:2606.31410 · Published 2026-06-30 · By Wanxia Cao, Chengzhen Duan, Pei Fu, Pengzhi Gao, Niu Lian, Fazhan Liu et al.

TL;DR

Xiaomi-GUI-0 addresses the persistent gap between the high benchmark performance of mobile GUI agents and their practical usability in real-world applications. Existing GUI agents are typically trained and evaluated on offline trajectories, simulated environments, or emulators, which fail to capture critical real-device factors such as account states, permission dialogs, payment authentication, and risk-control mechanisms that reshape interaction dynamics and cause abnormal states. Xiaomi-GUI-0 introduces a native end-to-end multimodal GUI agent trained and evaluated through a real-device-dominant hybrid infrastructure combining physical devices (smartphones, tablets, in-vehicle cockpits) with auxiliary sandboxes. This infrastructure collects realistic multi-source training data covering high-frequency tasks, long-tail intents, and model capability enhancements like reflection and memory.

The training leverages an error-driven data flywheel that converts failure trajectories from real rollouts into corrected actions, error explanations, and recovery demonstrations. The three-stage training pipeline integrates supervised fine-tuning, step-level reinforcement learning, and agentic reinforcement learning to incrementally develop basic execution, local corrections, and long-horizon planning with error recovery. Evaluation on the newly proposed RealMobile benchmark (real devices, live apps, real user states including abnormal conditions) and the public AndroidWorld benchmark shows Xiaomi-GUI-0 achieves 72.0% success on RealMobile and 78.9% on AndroidWorld, with substantial improvements in task execution stability and abnormal state handling under real conditions.

Key findings

• Xiaomi-GUI-0 achieves a 72.0% end-to-end task success rate on the real-device RealMobile benchmark.
• On the AndroidWorld simulated benchmark, Xiaomi-GUI-0 reaches 78.9% task success, outperforming baselines not evaluated in real settings (exact comparisons not specified).
• The error-driven data flywheel collects approximately 5,000 samples covering 14 abnormal state types including expired logins, captchas, payment authentication, permission prompts, which improves robustness to real-world abnormalities.
• The hybrid infrastructure runs on hundreds of physical devices from nearly 10 mainstream brands covering smartphones, tablets, and car cockpits, alongside hundreds of sandbox environments for scalable annotation and data collection.
• The multi-source training dataset comprises: high-frequency task data for common user operations, high-generalization data synthesized through a 5-level function tree and behavior buckets for long-tail intents, and agent-capability enhancement data incorporating reflection and memory.
• The three-stage training pipeline progressively develops capabilities: supervised fine-tuning for basic UI actions, step-level RL for granular action optimization and error identification, and agentic RL for long-horizon planning and error recovery.
• Interactive annotation and a teacher-model takeover mechanism enable precise labeling of erroneous steps and produce recovery demonstration trajectories, strengthening the model’s self-correction and abnormal-state recognition.
• RealMobile benchmark evaluation includes fine-grained sub-goal scoring and executable verification rules, supporting detailed performance measurement under realistic application, account state, and network conditions.

Threat model

The primary challenge addressed is the natural environmental variability and abnormal states encountered by mobile GUI agents during real-world task execution, caused by dynamic account conditions, permission and authentication dialogs, risk control interventions, and network variability. The model must robustly recognize and recover from these states to maintain stable operation. There is no explicit adversarial attacker; instead, the threat model captures system and environment-induced failure modes inherent to production mobile application usage.

Methodology — deep read

The paper’s core methodology is structured around a real-device-centric training and evaluation pipeline aimed at closing the gap between simulated study and real-world deployability for mobile GUI agents.

1. Threat Model & Assumptions: The adversary is not a threat actor but rather the distribution shift and environmental uncertainty in real mobile applications that produce abnormal states such as permission dialogs, captchas, payment authentication, expired sessions, and risk control interventions. The model must interpret screen observations, execute interface actions (tap, swipe, text input), and recognize/recover from abnormal conditions in a dynamic, real device environment.

2. Data: Training data originates from multiple sources collectively addressing head (high-frequency) user tasks, long-tail intents, and agent capability enhancement. High-frequency data is manually annotated from real user queries across multiple device types and applications, including abnormal state scenarios (around 5,000 samples covering 14 anomaly types). High-generalization data is constructed automatically using a 5-level function tree per app, behavior buckets representing user intents, query synthesis, rollouts on the hybrid infrastructure, and extensive trajectory cleaning (step- and trajectory-level). Agent capability data encodes chain-of-thought style reasoning into fields like observation, reflection, planning, decision, and memory. The error-driven data flywheel iteratively refines training data by replaying failure trajectories, annotating error steps, supplying correction rationale, and generating recovery demonstrations via a teacher model.

3. Architecture & Algorithm: The agent is a vision-language model-based multimodal policy that processes screen observations and instructions to output discrete interface actions defined in a unified action space (tap, swipe, text entry, navigation). Although exact model details are not deeply specified in the excerpt, the training involves supervised fine-tuning, followed by step-level reinforcement learning optimizing fine-grained action rewards, and finally agentic RL for long-horizon planning and error recovery with reflection and memory modules.

4. Training Regime: The three-stage progressive training pipeline begins with supervised fine-tuning on aggregated clean trajectories enabling fundamental interface control. Step-level RL employs group-relative reward shaping to learn fine-grained corrections and error identification. Agentic RL occurs in live or near-live environments, fostering sophisticated abilities like termination decisions, error reflection, trajectory recovery, and long-term planning. Specific hyperparameters, epochs, batch sizes, or hardware are not detailed in the provided text.

5. Evaluation Protocol: Evaluation is conducted on both the public AndroidWorld simulated benchmark and the newly constructed RealMobile benchmark. RealMobile emphasizes real-device deployment with authentic applications, real account states, network conditions, and abnormal scenarios. It uses fine-grained sub-goal scoring and executable verification rules to measure end-to-end completion, robustness to abnormalities, and task adaptability across multi-application contexts. An automatic evaluation pipeline replays trajectories to verify correctness. Results include success rates and robustness metrics. Ablations and statistical rigor details are not explicitly mentioned.

6. Reproducibility: The authors provide a publicly accessible project homepage, but the paper does not explicitly state if code or models are open-sourced. The dataset includes proprietary user queries, real-world trajectories on physical devices, and synthesized data, making full reproduction challenging without infrastructure access.

Example end-to-end flow: A real user query targeting a head function (e.g., “add to cart” in an e-commerce app) is annotated into multiple trajectories starting from various pages including intermediate and cross-app starting points. These trajectories are executed on real devices in the hybrid infrastructure, generating screenshots, steps, and abnormal states logs. Failures from the agent’s rollout on real devices are interactively labeled by annotators using the error-driven flywheel — locating error steps, supplying corrected actions and explanations. Teacher model takeover generates recovery demonstrations. These samples are incorporated back into training pipelines, progressively improving the model’s ability to perform this task robustly across app states and account conditions. At evaluation, the agent executes tasks on physical phones and tablets, with success and exception recognition scored by RealMobile benchmarks.

Technical innovations

• A real-device-dominant hybrid infrastructure combining physical mobile devices and sandboxes to collect realistic interaction data and support deployment-representative training and evaluation.
• An error-driven data flywheel that focuses on failure modes identified during real-device rollouts, generating corrected actions, reflective rationale, and recovery trajectories for targeted data augmentation beyond conventional data scaling.
• A progressive three-stage training pipeline integrating supervised fine-tuning, step-level reinforcement learning for fine-grained corrections, and agentic reinforcement learning enabling long-horizon planning and error recovery in real or near-real deployment conditions.
• Construction of the RealMobile benchmark, a real-device task execution benchmark that evaluates mobile GUI agents under authentic apps, genuine account states, network conditions, and a rich set of abnormal states with fine-grained sub-goal scoring and executable task verification.

Datasets

• High-frequency task data — approx. thousands of annotated trajectories including 5,000 abnormal state samples — collected from real user instructions on physical devices, proprietary.
• High-generalization data — synthesized from function trees (five-level hierarchical maps), behavior buckets, and automated query synthesis for long-tail intents — proprietary, augmented with manual and automated cleaning.
• Agent-capability enhancement data — structured chain-of-thought reasoning samples for reflection and memory — proprietary.
• RealMobile Benchmark — evaluation dataset recorded on hundreds of physical phones and tablets across 100 mainstream apps and 20 tablet/cockpit apps with real user traffic and abnormal states — proprietary.
• AndroidWorld Benchmark — public simulated benchmark for mobile task execution.

Baselines vs proposed

• Baseline unspecified on RealMobile: Xiaomi-GUI-0 task success = 72.0%
• Baseline unspecified on AndroidWorld: Xiaomi-GUI-0 task success = 78.9%
• Error-driven data flywheel augmentation leads to measurable improvements in robustness and abnormal-state handling (exact baselines unspecified)
• Use of physical-device hybrid infrastructure improves evaluation fidelity compared to emulator or simulation-based baselines (qualitative claim)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.31410.

Fig 1: Overview of our hybrid infrastructure. Hundreds of physical phones and dozens of physical tablets form the

Fig 2 (page 5).

Fig 3 (page 5).

Fig 4 (page 5).

Fig 5 (page 5).

Fig 6 (page 5).

Fig 7 (page 5).

Fig 8 (page 5).

Limitations

• Lack of detailed architecture and hyperparameter specifications limits reproducibility and direct comparisons.
• Proprietary real-user data and real-device infrastructure may limit accessibility and external validation.
• No explicit ablation studies or statistical tests reported for training stages, data tiers, or error-flywheel components.
• Unclear performance comparison with existing state-of-the-art GUI agents on identical benchmarks under matched conditions.
• Evaluation limited to Android ecosystems; iOS or cross-platform mobile environments not discussed.
• Security adversarial robustness (against malicious input or attack) is not evaluated; focus is on usability and robustness to benign environmental states.

Open questions / follow-ons

• How do various components of the three-stage training pipeline each quantitatively contribute to robustness improvements? Detailed ablation needed.
• Could the hybrid infrastructure and error-driven data flywheel approach be adapted to support other GUI platforms such as iOS or desktop OSes?
• What are the limits of scalability for the real-device infrastructure and error-flywheel feedback loop for continuous deployment in commercial settings?
• How resilient is the agent against adversarial or intentionally obfuscated UI states or malicious input designed to thwart automation?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, Xiaomi-GUI-0 presents a novel approach to improving GUI agent reliability and robustness in the wild, especially under conditions involving unpredictable abnormal states common in real mobile apps. The error-driven data flywheel and the real-device evaluation methodology highlight the importance of integrating failure-mode feedback and authentic environment factors into training pipelines. This is critical for systems expected to interact with CAPTCHAs, permission dialogs, and multi-step authentications that act as de facto bot-defense components.

Practitioners can view Xiaomi-GUI-0 as an example of moving beyond simulation to real environment validation, emphasizing a closed-loop system that not only trains on successes but focuses heavily on error diagnosis and recovery. While not focused specifically on bot detection or CAPTCHA solving, the principles of reflecting real-world distribution shifts, abnormal state recognition, and recovery are highly pertinent to designing resilient automated interaction systems and understanding the limits and failure scenarios of such agents in the presence of deliberate bot defenses.

Cite

@article{arxiv2606_31410,
  title={ Xiaomi-GUI-0 Technical Report },
  author={ Wanxia Cao and Chengzhen Duan and Pei Fu and Pengzhi Gao and Niu Lian and Fazhan Liu and Hui Liu and Heng Qu and Qinzhuo Wu and Zhehao Yu and Tongbo Chen and Shiqi Cui and Anan Du and Shukai Jia and Yuanfa Li and Wei Liu and Yike Liu and Wenchao Lu and Zhenbo Luo and Haoyuan Sun and Jiatong Sun and Cheng Tan and Yajie Wang and Changqiao Wu and Tao Xiong and Jiahui Yang and Yuxuan Yuan and Ruoceng Zhang and Shaojie Zhang and Jian Zhu and Jian Luan and Cong Zou },
  journal={arXiv preprint arXiv:2606.31410},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.31410}
}

Read the full paper

• arXiv:2606.31410 (abstract)
• arXiv:2606.31410 (PDF)