AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agents

Source: arXiv:2608.05891 · Published 2026-08-06 · By Weikai Xu, Yunren Feng, Haoxiang Lei, Kun Huang, Yuxuan Liu, Kang Zhao et al.

TL;DR

This paper addresses the challenge of building reliable, scalable GUI world models for mobile app agents that interact through pixel perception and touch actions. Prior approaches relying on either simulator-based environments or text/image/code-based world models suffer from issues such as unstable next-screen generation, incomplete modality coverage, and inconsistent action-transition logic. The authors propose AppDeltaWorld, a transition-grounded delta code world model that predicts the next GUI state as a reachable code update conditioned on an action-transition constrained retrieval of structural HTML references, combined with executable HTML generation and image-slot diffusion rendering. This hybrid approach promotes stable, high-fidelity screen generation that respects real app transition logic. Evaluation on CMGUIBench-500 demonstrates substantially improved layout and UI element fidelity compared to image-only and code-only baselines. As a training environment, AppDeltaWorld enables filtered closed-loop rollout generation improving an 8B-parameter agent (AppDeltaAgent) that achieves state-of-the-art action prediction and better long-horizon task success rates on benchmarks such as AndroidLens, MobileGym, and MobileWorld. Additional test-time reinforcement learning using the world model further boosts policy performance without requiring real app interactions.

Key findings

• AppDeltaWorld achieves a 73.51 overall score on CMGUIBench-500 under Code2World evaluation, outperforming image-only baselines like GPT-Image-2 (72.34) and code-only baselines such as Qwen3-8B (50.53), with a 45.5% relative improvement.
• In ablation tests, removing diffusion-based image synthesis reduces overall fidelity from 73.51 to 70.91; removing the transition-grounded retrieval (RAG) causes a larger fidelity drop to 67.46; removing both yields 65.68.
• AppDeltaAgent trained with AppDeltaWorld rollouts improves Action Matching Score (AMS) on AndroidLens from 80.33 to 90.28 and Average Task Progress (ATP) from 34.96 to 46.63 on low-level instructions, a 33.4% relative ATP gain.
• On MobileGym, AppDeltaAgent achieves 14.1% success rate, a 38.2% relative improvement over the 10.2% baseline Qwen3-VL-8B-Instruct.
• On real-device MobileWorld, AppDeltaAgent achieves 14.9% GUI-only success rate, improving 58.5% over the baseline (9.4%) and increasing average steps from 24.8 to 30.1.
• World-model-in-the-loop rollout construction filters and accepts only consistent trajectories where the world model rejects unreachable or invalid transitions, enhancing training data quality.
• Test-time world-model-based reinforcement learning further improves agent policy without requiring new real app interactions.
• Hybrid multimodal generation combining semantic text, executable HTML code, and diffusion-based image synthesis achieves better action-transition consistency and visual fidelity than single-modality approaches.

Threat model

The adversary is assumed to be an autonomous or semi-autonomous agent attempting to interact with mobile applications via pixel-level observations and discrete GUI actions without internal app APIs. They cannot modify app code or underlying system state but might generate invalid or unreachable GUI states if unconstrained. The model assumes access only to app UI observations and action commands and does not allow privileged app instrumentation or oracle information.

Methodology — deep read

The authors design AppDeltaWorld to overcome three main challenges for GUI world models: 1) stable fidelity in next screen generation, 2) hybrid modality covering structural UI code and visual assets, and 3) transition logic consistency to avoid invalid states.

Threat model assumes a mobile GUI agent needing to predict reachable next screens from current screenshots and discrete touch action commands. The agent cannot directly access app internals or APIs, but must produce realistic next UI states that obey app transition constraints.

Data provenance: The model is trained on a mixture of 100,149 GUI transition steps compiled from CMGUI, CAGUI, Magic-RICH, and ChiM-Nav datasets. Screens are reverse-engineered into executable hierarchical HTML representing structural layout (Level-1) and detailed interface (Level-2). App-level clustering organizes screen states into coarse functional categories and fine-grained clusters. Action histories and semantic screen text are extracted.

Architecture: The model operates in a retrieval-augmented generation (RAG) pipeline. First, the current screen’s structured textual state representation is vectorized and localized to a source cluster. A stored app-specific transition index enumerates reachable action-target clusters for the given action type and coordinates, constraining next-state retrieval. The Level-1 HTML reference is retrieved as a layout template. Then, a multimodal generator conditioned on the current screenshot, action, predicted next-screen text, and retrieved HTML produces a delta Level-2 HTML update (modifying only the changed UI components). A text-to-image diffusion model fills in image slots for visual assets difficult to describe in code. Finally, a browser renderer produces the next screenshot.

Training regime uses seeded real screenshots and instructions from GUI-Owl and OpenMobile to generate filtered synthetic trajectories via closed-loop rollouts inside the world model. An action policy model generates next actions, which the world model predicts and renders as observations, allowing supervised fine-tuning (SFT) of both models. The rollout data is quality-filtered to reject invalid or repetitive transitions.

Evaluation metrics include functional-logic accuracy (Sad, Sid), visual fidelity (Sele, Slay, SigLIP, DINOv2) on CMGUIBench-500, action matching score (AMS), average task progress (ATP) on AndroidLens, and success rates in MobileGym and MobileWorld benchmarks. Ablations demonstrate the contributions of both retrieval grounding and diffusion modules.

Reinforcement learning experiments use self-scoring rollouts generated by AppDeltaWorld to adapt policies without real data.

Code and evaluation data for CMGUIBench-500 are publicly released via Hugging Face. Exact replication requires access to these datasets, pretrained weights, and the transition-index memory.

Technical innovations

• Transition-grounded hierarchical HTML retrieval constructs reachable Level-1 reference code to guide next-screen generation, ensuring transition logic consistency.
• Hybrid multimodal world model combining semantic text prediction, delta executable HTML generation, and diffusion-based image-slot synthesis for richer UI rendering.
• Action-transition constrained retrieval filtering invalid next states to avoid hallucinated, unreachable GUIs during generation and data rollout.
• World-model-in-the-loop rollout construction creates closed-loop filtered synthetic trajectories from real seeds, improving policy supervised fine-tuning.
• Test-time reinforcement learning using the world model enables policy adaptation without any additional real-world app interaction.

Datasets

• CMGUI — 95,614 steps — public benchmark with reverse-engineered app HTML
• CAGUI — 2,978 steps — public GUI interaction dataset
• Magic-RICH — 1,304 steps — public mobile app interaction
• ChiM-Nav — 253 steps — navigation interaction dataset
• GUI-Owl (sampled 10%) — 56,237 steps — used for action model training
• AppDelta (generated) — 33,133 steps — generated rollouts using AppDeltaWorld
• OpenMobile — 27,360 steps — public mobile interaction dataset

Baselines vs proposed

• GPT-Image-2 (image generation): overall fidelity 72.34 vs AppDeltaWorld 73.51
• Gemini-3.1-Pro-Image (image generation): overall 71.57 vs AppDeltaWorld 73.51
• Qwen3-8B (code generation): overall 50.53 vs AppDeltaWorld 73.51
• AppDeltaAgent vs Qwen3-VL-8B on AndroidLens Total-LL AMS: 90.28 vs 80.33
• AppDeltaAgent vs Qwen3-VL-8B on AndroidLens Total-LL ATP: 46.63 vs 34.96
• AppDeltaAgent vs Qwen3-VL-8B-Instruct on MobileGym SR: 14.1% vs 10.2%
• AppDeltaAgent vs Qwen3-VL-8B-Instruct on MobileWorld GUI-only SR: 14.9% vs 9.4%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2608.05891.

Fig 1: Poor fidelity (upper) and inconsistent action inter-

Fig 2: Overview of AppDeltaWorld: (A) original mobile pages are reconstructed as HTML to train the world model; (B)

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• The world model training data is dominated by CMGUI (95%), potentially limiting generalizability to less represented apps or UI types.
• Though improved, success rates remain low in long-horizon real-world benchmarks like MobileWorld (≈15% SR), indicating much room for growth.
• The action-transition retrieval index depends on app-specific datasets and offline clustering, which may be costly to construct or maintain for new or evolving apps.
• Image-slot diffusion synthesis, while improving visual fidelity, may introduce inconsistencies not fully captured by existing metrics.
• The policy improvements from synthetic rollouts depend on stringent filtering; failure modes in rollout quality (e.g. loops, hallucinations) affect downstream performance.
• No explicit adversarial or robustness evaluation against intentional exploit or spoofing attempts on the world model was reported.

Open questions / follow-ons

• Can AppDeltaWorld scale to cover hundreds or thousands of diverse apps beyond current dataset distributions while maintaining stable fidelity?
• How does the world model perform under malicious or adversarial action inputs, and can it detect or reject purposeful invalid state transitions?
• Would fine-grained pixel-level consistency metrics or user-centric usability evaluations provide deeper insight into world model quality?
• How effective is the approach for very long-horizon interactive tasks involving conditional branching or multiple concurrent UI components?

Why it matters for bot defense

For bot-defense and CAPTCHA developers, AppDeltaWorld presents an advanced methodology for modeling and predicting complex GUI states on mobile apps through hybrid code retrieval and generation anchored in action-transition constraints. This approach ensures realistic next-screen predictions that are reachable in actual app logic, addressing challenges of state hallucination and unstable rendering that can complicate automated interaction modeling. By combining structured HTML with image-slot diffusion, it balances interpretability and visual fidelity, which is critical for evaluating bot behavior that tries to mimic genuine user interactions.

Practitioners building test environments or simulators for bot behavior analysis could leverage such transition-grounded world models to generate high-quality synthetic GUI interactions without requiring full simulator engineering or access to private app data. The closed-loop rollout construction enables generating realistic trajectories useful for training bot detection or behavior prediction models under controlled conditions. Moreover, the action-transition filtering mechanism could inspire CAPTCHA designs that integrate dynamic interface constraints to detect invalid action sequences characteristic of automated bots. Overall, AppDeltaWorld contributes a promising direction for scalable, stable GUI state modeling bridging symbolic structure and visual realism that bot-defense research can incorporate for more robust interaction simulations and user behavior understanding.

Cite

@article{arxiv2608_05891,
  title={ AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agents },
  author={ Weikai Xu and Yunren Feng and Haoxiang Lei and Kun Huang and Yuxuan Liu and Kang Zhao and Xiaolin Hu and Shuo Shang and Bo An },
  journal={arXiv preprint arXiv:2608.05891},
  year={ 2026 },
  url={https://arxiv.org/abs/2608.05891}
}

Read the full paper

• arXiv:2608.05891 (abstract)
• arXiv:2608.05891 (PDF)