Empowering GUI Agents via Autonomous Experience Exploration and Hindsight Experience Utilization for Task Planning

Source: arXiv:2606.27330 · Published 2026-06-25 · By Tianyi Men, Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao

TL;DR

The paper tackles the challenge of enhancing task planning abilities in small, open-source multimodal large language models (MLLMs) used by GUI web agents. These agents automate repetitive GUI tasks but struggle with planning complex, multi-step tasks and generalizing across diverse, unseen websites. The authors propose Planning Experience Exploration and Utilization (PEEU), a two-stage autonomous framework that lets agents explore unfamiliar web environments to generate high-level, hindsight-aligned task and trajectory data. PEEU synthesizes well-aligned task-trajectory pairs by extracting atomic experiences and aggregating them into stricter, high-level training tasks. This contrasts with prior approaches training only on atomic or coarse tasks with less alignment, which limits generalization. They also introduce the Task Decomposition Hierarchical Analysis Framework (TDHAF) to systematically evaluate compositional generalization across task granularities and domains.

Quantitative experiments on the real-world WebVoyager benchmark over seven unseen websites reveal significant gains: the 7B parameter model trained on PEEU-synthesized data achieves 30.6% accuracy, substantially exceeding the baseline Coarse-SFT 19.0% and outperforming a much larger 32B baseline model at 22.7%. The analysis with TDHAF shows that low-level atomic skills do not translate to high-level planning competence, while high-level task training facilitates stronger downward in-domain generalization and out-of-domain multi-level generalization. Overall, PEEU demonstrates that autonomous experience exploration combined with hindsight utilization to create better aligned and constrained training data is key to unlocking robust planning capabilities and cross-website generalization for smaller MLLMs.

Key findings

• The PEEU-SFT method using 2k trajectories with Qwen2.5-VL-7B achieves 30.6% overall accuracy on WebVoyager, significantly outperforming Coarse-SFT baseline at 19.0% and Qwen2.5-VL-32B Instruct at 22.7%.
• High-level task training yields a coverage percentage of up to 51.9% (7B model in-domain) and 37.8% (out-of-domain), compared to 9.1% and 25.7% for low-level atomic task training respectively, demonstrating better generalization.
• Atomic-level training on low-level tasks achieves above 80% accuracy on low-level tests but only 9.1%-18.8% accuracy on corresponding high-level task tests, confirming weak bottom-up transfer.
• Direct training methods (SFT, GRPO) substantially outperform retrieval-based prompting approaches for small models, e.g., 7B GRPO with PEEU reaches 19.9% vs 3.7% with atomic-level retrieval prompting.
• Trajectories synthesized with hindsight experience have stricter constraints and better alignment with tasks, reducing mismatch problems in coarse task data.
• Across various model scales (3B, 7B) and data amounts (0.1k, 2k trajectories), PEEU consistently improves OOD planning generalization on multiple unseen websites.
• PEEU utilizes autonomous goal-driven exploration via MLLM (GPT-4o) over 15 steps per task to build exploration trees capturing trajectories from diverse websites.
• The proposed TDHAF framework distinguishes three task granularities (low, mid, high) and systematically analyzes in-domain bottom-up, top-down, and out-of-domain multi-level generalization.

Threat model

The adversary is effectively the environment presenting unknown, unseen websites with diverse, partially observable GUI layouts and interaction rules. The agent must autonomously explore and learn without prior knowledge about the target web domains. The adversary does not actively interfere with agent observations or actions (no active adversarial sabotage), but the partial observability and content diversity poses a challenge. There is no assumption the adversary can manipulate the agent's internal states or training data.

Methodology — deep read

The paper's methodology centers on the Planning Experience Exploration and Utilization (PEEU) framework designed to enhance generalization in GUI task planning for web agents. The threat model targets scenarios where agents must autonomously explore unfamiliar websites to learn generalizable planning policies without human intervention or task-specific supervision.

1. Threat Model & Assumptions: The adversary is an unknown, unseen website environment presenting partially observable states with heterogeneous web page content and interaction dynamics. The agent has no prior knowledge of the new target domain and must discover task-relevant experiences autonomously. The agent assumes the ability to interact with the GUI through atomic actions like clicks and typing, and to observe page states visually.

2. Data: The framework begins with minimal input — the URL of a target website. An initial exploration phase uses the MLLM-based model (GPT-4o for exploration) to generate a diverse set of task goals tailored to the site content. The agent then performs autonomous, goal-driven interaction sequences up to 15 steps per task, building directed exploration trees (trajectories) from homepage to terminal states. The paper uses 0.1k and 2k trajectories derived from sources like Allrecipes and an unseen website. Seven other websites serve as held-out test domains. All trajectories include paired state-action transitions with before-after visual observations and are annotated through an experience extraction process.

3. Architecture & Algorithms: The approach consists of two main modules: (a) Planning Tree Exploration — the agent uses MLLM M to generate task lists from the homepage state, then recursively interacts with the environment to form an exploration tree R=(V,E) of states and actions capturing observed trajectories from task-driven rollouts. (b) Planning Experience Utilization — the agent extracts atomic experiences ϵ_t by comparing before-action and after-action states for each transition using the MLLM M. It aggregates sequences of atomic experiences µ into hindsight-aligned, constrained high-level tasks ˜d via a learned mapping Φ, thus generating a refined training dataset of aligned task-trajectory pairs.

Training the agent's policy π(s_t, h_t, ˜d)→a_t involves supervised fine-tuning (SFT) and goal-conditioned reinforcement learning via GRPO on the generated high-level aligned data. This trains a mapping from state observation and task description to next action.

4. Training Regime: Training uses Qwen2.5-VL-3B and Qwen2.5-VL-7B instructed models. For SFT, batch size 16, learning rate 5e-6, 5 epochs; for GRPO, batch size 20, learning rate 1e-6, 7 epochs. Training is performed on 4 NVIDIA A800 GPUs. Training uses llama-factory and verl frameworks. To ensure fairness, all methods use identical data amounts (0.1k or 2k trajectories) and model scales.

5. Evaluation Protocol: Models are evaluated on the WebVoyager benchmark with seven held-out websites across multiple categories. The metric is trajectory-level success rate (Step SR), measuring exact matching of predicted actions to ground truth trajectories. The proposed TDHAF framework is applied to systematically assess compositional generalization in three dimensions: in-domain bottom-up (low→high level), in-domain top-down (high→low level), and out-of-domain multi-level generalization (across unseen websites and tasks). Baselines include atomic-level and coarse-level methods, and prompt-based retrieval approaches. Ablations test task granularity, model size, data scale, and training method.

6. Reproducibility: The paper gives extensive experimental details with reported hyperparameters, datasets, and metrics. The usage of publicly known models (Qwen2.5-VL) and WebVoyager benchmark allows partial reproducibility, although it is unclear if code or datasets are publicly released.

Example Walkthrough: Given a URL, the MLLM exploration model generates a task list (e.g., find vegan recipes, casserole recipes under 1 hour). For each task, the agent explores the website up to 15 steps forming an exploration tree. Transitions are recorded as state-action pairs. Atomic experiences are extracted by analyzing states before and after each action. These are aggregated by MLLM hindsight to high-level tasks better aligned to the actual trajectory outcomes (e.g., adjusting recipe criteria to match actual discovery). The refined task-trajectory pairs form supervised training data. The policy model is fine-tuned on this data to improve planning generalization, tested on held-out websites.

Technical innovations

• Introducing PEEU, a two-stage autonomous method combining goal-driven exploration and hindsight experience aggregation to synthesize strictly aligned, high-level task-trajectory training data for GUI agents.
• Leveraging inverse hindsight alignment to recast exploration trajectories into better constrained training pairs, overcoming trajectory-task mismatches inherent in prior coarse task-based training.
• Proposing the Task Decomposition Hierarchical Analysis Framework (TDHAF) to quantitatively evaluate compositional generalization across three granularity levels (low, mid, high) and domains (in/out-of-domain).
• Demonstrating that high-level task training enables superior downward (top-down) in-domain generalization and multi-level out-of-domain generalization compared to atomic-level training.
• Empirically validating that direct training with PEEU data outperforms retrieval-based prompting for small multimodal LLMs in complex task planning.

Datasets

• WebVoyager — multimodal benchmark with seven real-world unseen websites — public (He et al., 2024)
• Allrecipes trajectories — 0.1k and 2k sampled autonomous explorations on cooking website — internal

Baselines vs proposed

• Qwen2.5-VL-7B Coarse-SFT: 19.0% overall accuracy vs PEEU-SFT 30.6% (+11.6%)
• Qwen2.5-VL-32B vanilla: 22.7% overall accuracy vs PEEU-SFT 30.6% (+7.9%)
• Atomic-SFT baseline (Qwen2.5-VL-7B, 2k traj): 21.7% vs PEEU-SFT 30.6% (+8.9%)
• Atomic-Prompt (Qwen2.5-VL-7B, 0.1k traj): 3.7% vs PEEU-GRPO 19.9% (+16.2%)
• Trajectory-Prompt (Qwen2.5-VL-7B, 0.1k traj): 3.7% vs PEEU-GRPO 19.9% (+16.2%)
• Coarse-GRPO (Qwen2.5-VL-7B, 2k traj): 19.0% vs PEEU-SFT 30.6% (+11.6%)
• Qwen2.5-VL-3B Atomic-SFT (2k trajectories): 16.7% vs PEEU-SFT 19.8% (+3.1%)
• Qwen2.5-VL-7B Atomic-GRPO (0.1k): 10.0% vs PEEU-GRPO 19.9% (+9.9%)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.27330.

Fig 1: The overview of planning experience explo-

Fig 2: An overview of planning experience exploration and utilization method with two stages.

Fig 3 (page 1).

Fig 4 (page 3).

Fig 3: This figure illustrates the task decomposition hierarchical analysis framework. The upper part shows

Fig 4: Generalization distribution pie chart for Qwen2.5-VL-3B. The table shows the distribution of eight types of

Fig 7 (page 6).

Limitations

• The datasets and trajectories used for training and evaluation are partially internal or not fully public, limiting reproducibility.
• The method relies on the availability of large oracle models (e.g., GPT-4o) during the exploration phase, which may be costly or inaccessible for some users.
• The exploration horizon is limited to short sequences (max 15 steps), which may restrict the complexity of learned tasks.
• No adversarial or intentional obfuscation attacks on exploration or planning were evaluated, so robustness under adversarial GUI environments is unclear.
• The approach presupposes environments where visual observations and atomic GUI actions can be effectively extracted and understood, which may not generalize to all GUI types.
• While compositional generalization is analyzed, long-term temporal dependencies or dynamics beyond task decomposition granularity are not thoroughly studied.
• Comparisons focus primarily on task granularity and training schemes under fixed data scale; effects of scaling data or models beyond 7B are not deeply explored.

Open questions / follow-ons

• How effective is PEEU under more complex, longer-horizon tasks or deeper exploration trees beyond 15 steps?
• Can the hindsight experience synthesis and alignment method scale to other modalities or GUI types beyond web navigation, e.g., mobile apps or desktop software?
• What is the robustness of PEEU-trained agents under adversarial GUI perturbations or intentional environment changes?
• How does the approach scale with larger base models (beyond 7B) or larger-scale exploration datasets? Do the gains saturate?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this paper provides insights into improving small multimodal agents' planning and task decomposition abilities via autonomous exploration and hindsight learning. Specifically, its findings highlight that training on well-aligned, high-level task data synthesized from real environment interactions yields stronger cross-domain generalization than relying solely on atomic or coarse task data. This is crucial when designing CAPTCHA or anti-bot challenges that target sequential, compositional interactions with web interfaces. Creating CAPTCHAs that require agents to perform multi-step plans across heterogeneous or unseen GUIs could exploit agents’ known weaknesses identified in low-level training scenarios.

Furthermore, the Task Decomposition Hierarchical Analysis Framework (TDHAF) proposed offers a structured way to evaluate agent generalization at multiple task granularities and domains. Bot-defense engineers could adapt this framework to assess if their CAPTCHAs effectively impede compositional generalization, thereby identifying potential weaknesses in agent planning abilities that can be used to harden defenses. The use of autonomy and hindsight experience synthesis also indicates that agents improve with environment-specific self-supervised learning, suggesting bot detection should consider dynamic obstacle or challenge generation to counter adaptive bots. Overall, the paper deepens understanding of how multimodal agents plan and generalize, informing the design of bot defenses relying on task complexity and cross-website variability.

Cite

@article{arxiv2606_27330,
  title={ Empowering GUI Agents via Autonomous Experience Exploration and Hindsight Experience Utilization for Task Planning },
  author={ Tianyi Men and Zhuoran Jin and Pengfei Cao and Yubo Chen and Kang Liu and Jun Zhao },
  journal={arXiv preprint arXiv:2606.27330},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.27330}
}

Read the full paper

• arXiv:2606.27330 (abstract)
• arXiv:2606.27330 (PDF)