From Bug Reports to Browser-Executable Procedures: An LLM-Driven Agent for Web GUI Bug Reproduction

Source: arXiv:2608.03598 · Published 2026-08-04 · By Cunming Zhang, Yu Pei, Michail Papadakis

TL;DR

This paper tackles the challenging problem of automatically reproducing web GUI bugs from natural-language bug reports, a critical step for debugging and maintenance. Existing techniques generally focus on code-level or mobile bug reproduction and do not support end-to-end execution and validation for dynamic web applications. To address this gap, the authors propose REBUG, an LLM-driven context-aware agent that reconstructs missing prerequisites from bug reports, generates a high-level reproduction plan, executes it by controlling a real browser with continuous state tracking, and validates the final browser state against expected and unexpected outcomes derived from the report.

REBUG’s two-stage approach separates prerequisite reconstruction from browser execution, leveraging structured state summaries and action histories to adapt to dynamic web GUIs. The system was evaluated on 667 real-world bug reports from four large open-source web applications, substantially outperforming baseline approaches in reproduction success rate (49.96%), task completion (74.96%), and action execution success rate (86.54%). A historical replay on restored buggy versions confirmed that many reconstructed procedures trigger the original failures. These results demonstrate that explicit context recovery and state-aware execution enable scalable automated web GUI bug reproduction directly from informal bug reports.

Key findings

• REBUG achieves an average reproduction success rate (RSR) of 49.96% over 667 real-world web GUI bug reports from Ghost, Metabase, NocoDB, and n8n.
• Mean task completion rate (TCR) of REBUG is 74.96%, indicating substantial partial progress even on challenging reports.
• Mean action execution success rate (AESR) of REBUG is 86.54%, reflecting precise control of browser interactions.
• Disabling the Context Builder significantly reduces reproduction effectiveness, confirming the importance of explicit prerequisite reconstruction (RQ2 ablation).
• Disabling the Plan Generator also decreases performance, showing the value of a high-level reproduction plan scaffold.
• Compared to baselines like Browser-use (no preparation) and direct LLM script generation, REBUG improves RSR by over 20 percentage points.
• Historical replay on 40 successful cases demonstrates that reconstructed procedures often reproduce the original bug-present behavior on restored buggy versions.
• Failure analyses reveal dominant challenges including missing prerequisites, dynamic UI changes, and brittle element targeting.

Threat model

The adversary is assumed to provide natural language bug reports that may omit necessary prerequisites like files, configurations, or entry points for reproduction, but cannot directly provide executable scripts or control the deployed application or browser internals. The system must operate under these partial and noisy reports, reconstruct missing context, and safely interact with a sandboxed, controlled browser environment. The adversary cannot compromise the evaluation environment or alter the browser agent’s code. The threat focuses on incomplete or ambiguous reporting rather than malicious manipulation.

Methodology — deep read

1. Threat Model & Assumptions: The adversary can provide natural language web GUI bug reports with incomplete prerequisites (missing input files, configs, entry points). The system assumes access to a real browser environment deployed with known open-source applications. The adversary cannot alter the deployed backend, cannot provide executed scripts, and cannot control runtime browser internals.

2. Data: The authors curated a dataset from GitHub issue trackers of four mature, actively maintained open-source web apps: Ghost, Metabase, NocoDB, and n8n. Starting from ~4,000 candidate bug reports filtered by keywords and labels, they manually confirmed 875 valid bug reports after filtering out false positives. Further refinement removed non-GUI failures or version-locked issues, yielding 667 bug reports with annotated expected and unexpected browser-observable outcome pairs (oracles). Each report preserves original natural language text and metadata. Additionally, a historical replay subset of 40 reports was created by restoring buggy app versions.

3. Architecture / Algorithm: REBUG consists of two stages: (a) Preparation Stage, which reconstructs missing execution context (files, configs, inputs) via a Context Builder agent that iteratively assesses prerequisite needs from the report and project artifacts, materializes files using a sandboxed executor with bounded operations, maintains an explicit artifact inventory and action history, and generates a high-level ordered reproduction plan capturing the main intent steps. (b) Execution Stage, which runs a stepwise execution loop driving a real Chrome browser. After each action (click, input, navigation, file upload, etc.), REBUG captures screenshots and structured DOM states, updates a compact browser-observable record (Information Center), grounds next actions in current state and history, and adapts on dynamic UI changes. An LLM Action Generator emits structured JSON actions validated against schemas. The system enforces timeouts and failure thresholds to terminate gracefully. The final Evaluator module compares the observed end state against the annotated outcome oracle using multi-modal evidence (DOM summary, URL, screenshots), returning a binary reproduction success with an evidence-based rationale.

4. Training Regime: This is a system integrating LLMs as agents rather than traditional ML model training. Experiments vary the backbone LLM (GPT 5 mini, Claude 4.5 Haiku, Gemini 2.5 Flash). The agents operate in a sandboxed environment but details on model fine-tuning or hyperparameter optimization are not specified.

5. Evaluation Protocol: Metrics include reproduction success rate (RSR) requiring completion and matching expected but not unexpected final states, task completion rate (TCR) measuring fraction of planned steps completed, and action execution success rate (AESR) indicating fraction of successfully executed low-level browser actions. The system is compared to baselines: a browser usage agent without preparation, and direct one-shot LLM scripted generation. Reporting is project-balanced, with multiple independent attempts per bug report. Statistical tests are not explicitly described. Historical replay validates whether reproduced procedures trigger original bug behavior on buggy versions.

6. Reproducibility: The authors build a new benchmark dataset with annotated bug reports and oracles, applied to publicly available open-source web apps deployed in standardized environments (Docker, local installs). The codebase and benchmark details were not explicitly stated as public but the benchmark design supports systematic future evaluation. The study includes detailed logs, traces, and failure analyses.

Example Walk-through: For a given bug report, REBUG’s Context Builder first analyzes missing prerequisite artifacts (e.g., required input files) and attempts iterative synthesis or reuse from local whitelisted files. Once prerequisites are ready, it generates a high-level browser interaction plan from the bug description. The execution loop navigates the browser, interacts via clicks and inputs mapped to identifiable DOM elements, updates internal browser state summaries after each action, and adapts next interactions accordingly. After finishing or terminating on failure thresholds, the Evaluator compares final URL, DOM snapshot, and screenshots against oracle conditions derived from the report text to judge success.

Technical innovations

• An LLM-driven context-aware agent system that iteratively reconstructs missing web GUI bug report execution prerequisites from sparse natural language reports and local artifacts.
• A two-stage approach separating prerequisite materialization with an artifact inventory from state-aware, stepwise browser execution driven by dynamic grounding in live DOM and UI states.
• A structured Information Center that maintains browser state records, action histories, and runtime feedback to enable adaptive web GUI reproduction in highly dynamic single-page applications.
• An integrated automatic outcome evaluator that judges reproduction success by comparing multi-modal browser-observable evidence (DOM, screenshots, URLs) to report-derived expected and unexpected descriptions.

Datasets

• GitHub Web GUI Bug Reports Benchmark — 667 confirmed bug reports — curated from Ghost, Metabase, NocoDB, and n8n GitHub issue trackers with annotated outcome oracles
• Historical Replay Subset — 40 successful reproduction cases — restored buggy application versions for replay validation

Baselines vs proposed

• Browser-use (no preparation): RSR = ~29% vs REBUG RSR = 49.96%
• Direct LLM script generation (GPT 5 mini): lower than REBUG by approximately 20+ percentage points in RSR
• REBUG with GPT 5 mini: RSR ~50%, TCR ~75%, AESR ~86%
• Disabling Context Builder module decreases RSR significantly (exact numbers not given)
• Disabling Plan Generator reduces reproduction effectiveness (performance drop details in paper)

Limitations

• The evaluation uses current deployed versions rather than restoring exact historical buggy environments for all reports, so RSR reflects procedural completion and consistency, not guaranteed bug-triggering behavior.
• Context reconstruction excludes complex prerequisites such as installation, build environments, or external private services, limiting reproduction applicability for infrastructure-dependent bugs.
• The approach depends heavily on LLMs and may be brittle to language ambiguity or report noise; the system’s reliance on accurate artifact inventories requires careful sandboxing.
• No explicit adversarial evaluation against adaptive attackers or manipulation of reports was conducted.
• Dynamic UI changes and brittle element selectors still cause execution failures, limiting full automation.
• The benchmark and system focus on open-source web apps with certain scale and maturity; generalization to proprietary or highly customized GUIs is unknown.

Open questions / follow-ons

• How can context reconstruction be better generalized to complex dependencies like backend services or multi-component integrations?
• What strategies can improve robustness to highly dynamic UI changes and brittle element selectors in evolving web apps?
• Can reproduction success be improved by integrating richer multimodal inputs such as video or clickstream logs attached to bug reports?
• How would the approach perform under adversarially crafted or intentionally obfuscated bug reports designed to mislead context reconstruction?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this work showcases advanced applications of LLM-driven agents to interpret and act upon natural language instructions in complex dynamic web environments. It highlights critical challenges in translating noisy, incomplete textual descriptions into reliable, automated browser interactions grounded in real-time UI state, which is relevant for developing robust bot detection or prevention systems that must adapt to unpredictable user inputs and evolving page layouts.

Moreover, REBUG’s approach of reconstructing missing execution context and validating end states against expected behaviors can inform CAPTCHA systems that verify genuine user workflows or detect scripted automation attempts. Practitioners could consider leveraging similar state-aware browser interaction frameworks combined with language models to improve bot vs human discrimination or to auto-generate reproducible security test cases mimicking attacker behaviors described in reports.

Cite

@article{arxiv2608_03598,
  title={ From Bug Reports to Browser-Executable Procedures: An LLM-Driven Agent for Web GUI Bug Reproduction },
  author={ Cunming Zhang and Yu Pei and Michail Papadakis },
  journal={arXiv preprint arXiv:2608.03598},
  year={ 2026 },
  url={https://arxiv.org/abs/2608.03598}
}

Read the full paper

• arXiv:2608.03598 (abstract)
• arXiv:2608.03598 (PDF)