Plover: Steering GUI Agents through Plan-Centric Interaction

Source: arXiv:2607.15193 · Published 2026-07-16 · By Madhumitha Venkatesan, Shicheng Wen, Jiajing Guo, Jorge Piazentin Ono, Liu Ren, Dongyu Liu

TL;DR

Plover addresses the challenges of autonomous GUI agents that operate over dynamic, visually complex interfaces where failures and drifts from user intent are common. Current vision-based multimodal agents execute highly flexible tasks but hide their planning and replanning processes internally, making it difficult for users to inspect or locally correct errors without restarting the entire workflow. Plover innovates by externalizing task plans as persistent, inspectable, and editable artifacts and structuring automation as a human-agent collaborative process. Its planner–executor architecture enables users to review evolving plans, intervene precisely through natural language or screenshot annotations, and trigger localized plan repairs while preserving previously completed steps.

Through a formative study and extensive evaluations on 38 benchmark tasks with 26 failure cases, Plover demonstrates that making plans visible and interventions local significantly improves recoverability of GUI automation. Specifically, 23 out of 26 failure cases improved with collaborative repair, with 17 becoming full successes. The system’s transparent replanning mechanisms offer better user situational awareness and control over execution drift, enabling effective failure recovery and reducing redundant work. Analyses reveal common failure modes like spatial ambiguity and execution drift that are amenable to guided repair using Plover’s approach.

Key findings

• Out of 38 challenging GUI automation tasks, 26 failed under autonomous execution; collaborative repair in Plover improved 23 of these.
• 17 failure cases recovered to full success after interventions averaging 2.04 per task.
• Persistent, editable plan artifacts enable localized corrections that preserve prior progress, reducing all-or-nothing rewrites.
• Multimodal annotation (screenshot markup) was consistently rated more precise for spatial corrections than natural language alone.
• System-driven Intelligent Replanning detected non-progress via combined behavioral loop detection and visual stability verification to autonomously trigger localized repair steps.
• Run Timeline visualization documenting plan evolution and execution history enhanced user understanding of plan changes and intervention effects.
• Failures often arose from spatial ambiguity in dense interfaces, execution drift across multi-step workflows, and misinterpretation of intermediate UI states.
• Explicit replanning surfaced as a transparent user-actionable process rather than a hidden model state update improved repairability and user trust.

Threat model

The adversary is environmental instability, dynamic UI changes, ambiguous and dense screen layouts causing autonomous agent errors and execution drift. The system does not consider active adversaries but focuses on robustness and recoverability against incidental failures due to interface variability, non-progress, and misinterpretation within vision-based GUI automation.

Methodology — deep read

1. Threat Model & Assumptions: Plover assumes an end user executing complex GUI tasks with a vision-based automation agent in a desktop environment. The adversary is implicit in nature: unstable or evolving GUIs cause the automation to drift from intended behavior, producing failure modes like clicking wrong controls or missing dialogs. The system does not address malicious adversaries but focuses on recoverability from environmental variability and agent inference errors.

2. Data: The evaluation leverages 38 benchmark GUI tasks representing complex workflows with dynamic states and failure factors. 26 tasks resulted in autonomous failure used for repair analysis. The dataset provenance originates from commonly used desktop applications and legacy enterprise tools, including multi-page forms and modal dialogs. Data includes screenshots, task plans, user annotations, execution logs, and system states. No public dataset release is indicated.

3. Architecture / Algorithm: Plover employs a Planner–Executor architecture implemented as separate services. The Planner Service uses large language models (LLMs) to generate structured, versioned plans from user prompts. Plans enumerate explicit GUI interaction steps. The Executor Service performs step grounding by mapping planned actions to concrete mouse/keyboard operations within a remote GUI environment (e.g., OS-level VM). Plans are externalized as editable artifacts, allowing inspection and revision via a rich Agentic Interface. User interventions trigger local plan edits communicated back to the Planner, rather than wholesale regeneration. Intelligent Replanning (IR) allows both User-Driven IR—via natural language guidance or screenshot-based multimodal annotations—and System-Driven IR—using non-progress detection through combined behavioral sequence loops and perceptual hash-based visual stability.

4. Training Regime: Not explicitly described, suggesting planner LLMs leverage pre-trained foundation models fine-tuned or prompted for GUI tasks. Execution and intervention processes are governed by heuristics and system policies rather than learned models.

5. Evaluation Protocol: Evaluation involves a formative qualitative study with six participants testing an early prototype to gather interaction challenges and validate design goals. Benchmark failure-case repair experiments quantitatively assess recovery rates, intervention counts, and success criteria comparing autonomous versus collaborative runs. Scenario-based workflow analyses employ visual similarity and plan alignment metrics to assess execution stability and plan drift. Failure archetypes are qualitatively categorized, and user feedback on correction mechanisms is collected.

6. Reproducibility: Code or model weights are not reported as released. The system operates on closed, proprietary GUI environments within controlled virtual machines. Method details are sufficient for conceptual replication but exact dataset and tooling are unavailable publicly.

Concrete Example: A user runs a multi-step form-filling task where the agent mistakenly inputs a wrong numeric field due to confusion with a visually similar widget. The user uses the Agentic Interface Multimodal Annotation tool to highlight the screenshot region corresponding to the correct target. This triggers User-Driven IR, producing a localized plan repair proposal that revises only the remaining plan steps linked to that input while preserving completed progress. The user reviews and approves the updated plan which the Executor then continues to run, now correctly filling the form without restarting the task.

Technical innovations

• Externalizing evolving GUI task plans as persistent, editable artifacts enabling explicit user inspection and localized revision during execution.
• A planner-executor split architecture with coordinated versioned plans to separate planning, execution, and repair transparently.
• Multimodal annotation-based interventions that constrain localized plan updates grounded in screenshot spatial context to resolve ambiguity beyond natural language instructions.
• Intelligent Replanning combining behavioral loop detection and perceptual hash-based visual non-progress identification to autonomously trigger transparent recovery steps.
• Run Timeline and branching provenance visualization that audibly tracks plan revisions, user interventions, and execution progression for mental model alignment.

Datasets

• Benchmark GUI automation task set — 38 tasks with multi-step workflows and dynamic interfaces — Internal/proprietary

Baselines vs proposed

• Autonomous execution baseline: 26/38 tasks failed; versus Plover collaborative repair: 23/26 failures improved, 17 full task success after ~2.04 interventions/task

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.15193.

Fig 1: The Plover system architecture.

Fig 2: Plover Agentic Interface. (a) System Status Bar shows execution phase and plan state. (b) Planner Chat supports

Fig 3: Provenance Bar visualizing branching plan revi-

Fig 4: Multimodal Annotation workflow for resolving

Fig 5: Representative failure archetypes and recovery pathways in Plover. Each row shows the autonomous failure state

Fig 6 (page 8).

Fig 7 (page 8).

Fig 8 (page 8).

Limitations

• No adversarial testing against malicious input or deliberate obfuscation attackers; focused on environmental variability only.
• Evaluation limited to 38 GUI tasks and 6 participants, restricting statistical generality and diversity of UI scenarios.
• No publicly released dataset, codebase, or pretrained weights provided to enable independent reproduction.
• Planner relies on large language models but training regimen, hyperparameters, and LLM architecture details are largely unspecified.
• User interventions depend on manual input; system-driven replanning can detect non-progress but cannot autonomously resolve complex semantic failures fully.
• Interface usability tested in small formative study only; real-world deployment scalability and multi-user collaboration not examined.

Open questions / follow-ons

• How can the planning and replanning components be optimized or learned end-to-end to further reduce failure rates and interventions needed?
• What are the best design patterns to scale plan-centric GUI automation for multi-user or distributed settings involving concurrent collaborative repair?
• How can Plover’s approach integrate with adversarial robustness techniques to defend against malicious manipulation of GUI states?
• Can richer multimodal signals (e.g., gaze, haptics) be incorporated to improve spatial grounding and reduce ambiguity in correction inputs?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, Plover’s approach emphasizes transparency and user steerability in long-horizon GUI task automation. Many bot and crawler failures arise from environmental variability and silent failure propagation analogous to GUI drift. Externalizing and versioning task plans as inspectable, editable artifacts could inspire new interaction paradigms for manual bot monitoring, targeted repair, and audit trails in automated bot workflows. Similarly, the use of multimodal spatial grounding to enable precise, local corrections may inform bot detection and mitigation interfaces that require fine-grained human input to adapt defenses dynamically. The system-driven non-progress detection combining behavioral and visual signals could offer heuristic insights for detecting bot stalls or evasion attempts. Overall, Plover’s conceptual framing as a collaborative repairable process rather than opaque autonomous action aligns with increasing industry interest in mixed-initiative systems balancing automation and human oversight in bot management.

Cite

@article{arxiv2607_15193,
  title={ Plover: Steering GUI Agents through Plan-Centric Interaction },
  author={ Madhumitha Venkatesan and Shicheng Wen and Jiajing Guo and Jorge Piazentin Ono and Liu Ren and Dongyu Liu },
  journal={arXiv preprint arXiv:2607.15193},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.15193}
}

Read the full paper

• arXiv:2607.15193 (abstract)
• arXiv:2607.15193 (PDF)