All Green, Still Broken: Real-Flow Verification Lessons from an LLM-Integrated, Multi-Market Web Application
Source: arXiv:2606.22475 · Published 2026-06-21 · By Muhammad Bilal, Ali Hassaan Mughal
TL;DR
This paper investigates why a large, continuously passing automated test suite failed to prevent production defects in a modern web application integrating large language models (LLMs), multi-market internationalization, and browser-driven front-ends. The authors conducted an in-depth analysis of all 252 bug-fix commits over six weeks in a production rental-search assistant, whose test suite grew to 1,553 cases and continually passed before deployment. Despite this, six user-facing defects reached production, including one recurring error. They identify that 44% of bugs escaped through four key boundaries (or seams) that component-level unit tests cannot cover: the live browser runtime, non-default market environments, end-to-end user flows, and whole-system behavior. The paper introduces a four-seam framework to classify defects by seam, provides a full census of defects from the project, and prescribes practical testing practices focused on these seams rather than expanding component tests.
Key findings
- The automated test suite comprised 1,553 test cases and passed on every change, yet 6 user-facing defects still reached production.
- Out of 252 bug-fix commits, 110 (44%) fixed defects that escaped at four seams unobservable by component-level tests: runtime (29 fixes), market (23 fixes), flow (35 fixes), and system (23 fixes).
- A recurring defect shipped twice due to missing a guard at the seam, demonstrating that fixes must add regression checks at the seam to prevent recurrence.
- Different seams contribute similarly to escaped defects; no single seam dominates, making seam-focused test strategies necessary across multiple boundaries.
- Component tests cover only 56% of defects (backend and data source changes), so test count and coverage are weak quality indicators.
- A scripted, real-browser end-to-end pass through the live product, especially on non-default markets, is the lowest-cost check that observes many seam defects.
- Mapping defects to seams is possible with an automated deterministic classifier on commit messages, enabling teams to prioritize test investment.
- Internationalization defects arise at the market seam due to assumptions baked into validators and configuration limited to default market testing.
Threat model
N/A—this is a software quality and testing empirical study rather than an adversarial security investigation. The focus is on unintentional defect escapes in production despite existing test suites, not on malicious attacks or adversaries.
Methodology — deep read
The study examined a production rental-search assistant developed over six weeks, ingesting listings from multiple external portals, applying large language model screening and drafting outreach, and offering a two-market service with different locales, currencies, and languages. The test suite grew to 1,553 cases across 144 files within 43 days, running on every code change and blocking deployment on failure. Yet six user-facing defects shipped after the test suite plateaued. The authors took a snapshot of the repository (740 commits) and extracted all 252 commits whose messages started with 'fix', representing a near 1:1 ratio of fixes to features. They developed a four-seam framework defining boundaries between component tests and production behavior: runtime (browser environment), market (non-default locale), flow (end-to-end user interactions), and system (whole-system rules and baselines). A deterministic classifier applied priority keyword rules to assign each bug fix to exactly one seam or to a general backend category, with no manual edits, producing a reproducible lower bound on seam defect share. They also recorded whether test files changed as a proxy for regression guard presence. The authors present 6 specific defect cases, each linked to one seam, illustrating failure modes unobserved by unit tests but exposed by live runtime, market differences, interaction flow, or system-level monitoring. The approach did not increase unit tests but instead shifted the definition of done toward scripted real runtime verification per changed screen, exercising the non-default market, and adding build-time checks preventing defect recurrence in guarded seams. Limitations include reliance on commit message keyword classification (potential mislabeling), single-application scope, and no direct adversarial or cross-project evaluation. Code and classifier rules are released on Zenodo for replication. Overall, the paper provides a deep empirical study linking defect escape routes to boundaries untracked in common test practices in modern web apps combining LLMs, multi-market internationalization, and browser front-ends.
Technical innovations
- Definition and empirical validation of a four-seam framework identifying runtime, market, flow, and system boundaries as key defect escape routes invisible to component tests.
- Development of a deterministic, keyword-based classifier mapping bug-fix commit messages to seam categories for full defect census without manual bias.
- Proposal of a low-cost test practice emphasizing a single scripted real browser end-to-end pass per changed screen, especially in non-default markets, to observe seams.
- Introduction of build-time guards that fail on defect recurrence at the seam, preventing shipping repeated bugs invisible to component test suites.
Datasets
- Project commit history snapshot — 740 commits total, 252 bug-fix commits used — private project repository
Baselines vs proposed
- Component-level unit tests: passed 100% of 1,553 test cases yet missed 44% of bug-fix defects observable only at seams
- Real browser runtime tests (added in response): caught runtime seam defects not detected by component tests
- End-to-end user interaction testing (added): caught flow seam defects invisible to isolated component tests
- Market-aware tests in non-default locales (added): uncovered 23 market seam defects missed in default-market only tests
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.22475.
Fig 1 (page 7).
Fig 2 (page 7).
Limitations
- Single application studied limits generalizability to other systems and architectures combining LLMs, internationalization, and browser front-ends.
- Classification of defect commits based solely on commit message keywords may mislabel some fixes or undercount seam defects.
- Analysis is retrospective and observational; no controlled experiments or adversarial tests were performed.
- The proportion of silent or unreported defects that escaped detection was not measured, potentially underestimating seam defect share.
- No cross-validation on external repositories to verify that the four-seam framework applies broadly.
- The impact of expanded testing on development cost or cycle time was not quantified.
Open questions / follow-ons
- How well does the four-seam framework generalize across other web applications, especially those without LLM integration or internationalization?
- Can automated instrumentation or monitoring tools detect seam-boundary conditions in real time to prevent defects?
- What is the cost-benefit trade-off of seam-focused real browser and end-to-end testing compared to expanding unit test coverage?
- How might adversarial testing strategies be integrated with seam-aware testing to improve security-relevant properties?
Why it matters for bot defense
For bot-defense and CAPTCHA teams, this paper highlights the risk that large automated test suites can miss defects recurring at boundaries between isolated tests and real user behavior or environment conditions. Similar to CAPTCHA systems integrating large models and serving multi-market audiences, unseen seams such as browser runtime differences, internationalization edges, user interaction flows, or whole-system rules can harbor subtle defects that evade component-level test coverage. Bot-defense engineers should consider end-to-end real environment testing, particularly exercising less common markets and genuine user workflows, to catch defects in CAPTCHA deployment or validation logic invisible to unit or API tests. The concept of adding build-time guards at these seams also offers a way to prevent repeated regressions at vulnerable integration points. Overall, the four-seam framework guides practitioners toward targeted testing investments that better reflect complexity in modern, dynamic multi-market, LLM-enhanced web systems.
Cite
@article{arxiv2606_22475,
title={ All Green, Still Broken: Real-Flow Verification Lessons from an LLM-Integrated, Multi-Market Web Application },
author={ Muhammad Bilal and Ali Hassaan Mughal },
journal={arXiv preprint arXiv:2606.22475},
year={ 2026 },
url={https://arxiv.org/abs/2606.22475}
}