Flow-A11y: Flow-Aware Accessibility Testing
Source: arXiv:2607.03100 · Published 2026-07-03 · By Nasr Eddine Fliti, Leisan Kokorina, Florian Tambon, Michael Papadakis
TL;DR
This paper addresses a critical gap in automated web accessibility testing: many important WCAG criteria depend not on static page snapshots but on dynamic, interaction-driven runtime behaviors during user flows. Common automated tools analyze static DOM states, missing dynamic accessibility failures such as keyboard traps or delayed live-region updates. To tackle this, the authors propose FLOW-A11Y, a flow-aware accessibility testing framework that executes natural-language defined user interaction scenarios in a real browser and records comprehensive runtime traces including DOM snapshots, accessibility tree states, keyboard events, focus changes, and more. It extracts criterion-specific evidence packets from these traces and gates judgments based on evidence availability before invoking LLMs fine-tuned with criterion-specific prompts to make grounded accessibility judgments. Evaluated on 19 real web scenarios spanning 45 dynamic WCAG criteria, FLOW-A11Y achieves over 10x higher agreement with oracle labels than a generic browser-agent audit. Its evidence calibration layers improve fail precision from 23.5% to 41.4% and eliminate invalid evidence citations. This demonstrates that runtime traces plus criterion-aware analysis are practical and effective for automating dynamic accessibility criteria that purely static tools cannot assess, substantially reducing the need for manual inspection.
Key findings
- FLOW-A11Y achieved 45.6% exact five-way status accuracy over 765 criterion assessments vs 3.3% for a generic browser agent audit (Fig 6).
- Fail precision improved from 23.5% for a naive baseline to 41.4% with FLOW-A11Y's evidence calibration.
- FLOW-A11Y eliminated invalid evidence-reference model outputs, reducing them from 35 to zero (Section VI).
- The benchmark covered 19 real public websites with 19 natural-language interaction scenarios targeting 45 dynamic WCAG 2.2 success criteria.
- Runtime traces include multimodal evidence: DOM snapshots, accessibility trees, keyboard and pointer events, focus transitions, live-region mutations, screenshots, and media states.
- Criterion-specific evidence gating prevents model invocation when required evidence is missing or the criterion is not applicable.
- LLM judgments are restricted to citing resolvable evidence fields; unsupported citations downgrade findings to 'INCOMPLETE'.
- Error analysis exposes precision-recall tradeoffs and limitations in scenario coverage impacting some criteria's detectability (Section VII).
Threat model
The adversary is the web application potentially exposing dynamic accessibility barriers triggered during user interactions. They do not control or evade the auditor's browser instrumentation but may implement faulty or incomplete accessibility features visible only during runtime user flows. The auditor assumes no adversarial evasion or counter-instrumentation attacks but seeks to detect dynamic failures manifest during scenario execution.
Methodology — deep read
The threat model assumes an auditor agent executing scripted scenarios on publicly reachable web applications without adversarial influence on the code or server responses. The adversary is thus a non-adaptive web app implementing accessibility barriers that only manifest during user interactions. FLOW-A11Y requires no assumptions about attacker knowledge or intent beyond these.
Data provenance stems from 19 publicly accessible websites hosting interactive UI patterns relevant to 45 dynamic WCAG 2.2 criteria. Each site was paired with a human-curated natural-language scenario specifying user goals (e.g., open menu, apply filter, submit form). The authors manually labeled oracle outcomes for all criterion-by-scenario pairs to create a ground truth benchmark covering 765 scored assessments from 855 total attempts. The dataset and code are publicly released for reproducibility.
The architecture centers on executing the defined scenario in a real browser. An agent performs each user step (clicks, typing, navigation), capturing ordered runtime states before and after each action. Each runtime state logs multimodal evidence: full DOM and accessibility tree snapshots, keyboard and pointer event histories, focus and active element info, live-region mutations, screenshots, media playback states, console output, timing signals, and DOM mutations. These collectively form the raw runtime trace.
Next, 9 dynamic WCAG probes specialized by criterion family transform the raw trace into criterion-specific evidence packets summarizing key signals like focus sequences for keyboard and focus criteria, validation feedback for forms, live-region announcement for status messages, etc. The evidential requirements per criterion are maintained in a registry.
The analyzer first gates each criterion, skipping any without evidence presence or applicability, thus returning 'INCOMPLETE' or 'NOT APPLICABLE' and avoiding spurious judgment calls.
Criteria passing the gate are sent, along with their evidence packet and WCAG criterion text, to an LLM prompted with a strict JSON contract. The LLM outputs JSON results containing pass/fail/warning decisions with concrete citations of evidence fields.
After LLM inference, outputs citing unresolved evidence references are downgraded to INCOMPLETE. Further, fail outputs citing only indirect evidence are downgraded from FAIL to WARNING.
This calibrated, evidence-gated pipeline produces five decision levels: confirmed fail, probable violation, weak signal, insufficient evidence, and not applicable.
Evaluation contrasted FLOW-A11Y to a naive browser-agent audit baseline on the benchmark, measuring exact status accuracy, fail precision and recall, and the incidence of invalid evidence references. Results showed FLOW-A11Y's runtime trace plus evidence-grounded LLM analysis substantially outperformed the baseline in detecting interaction-dependent criteria.
The workflow clearly separated scenario execution from criterion analysis, enabling reuse of runtime traces for multiple criteria. Scenarios were designed to exercise relevant interactions with real sites rather than synthetic or toy examples. Artifact packages with runtime evidence, prompts, and outputs were exported for auditability and reproducibility.
Overall, the methodology rigorously connects user-interaction-driven runtime evidence collection, dynamic criterion-specific evidence construction, evidence-gated LLM reasoning with grounded outputs, and comprehensive benchmark evaluation on realistic web applications.
Technical innovations
- Representing accessibility assessment as runtime interaction traces combining structural, visual, behavioral, and state-change evidence rather than static snapshots.
- Dynamic WCAG probes extracting criterion-family-specific evidence packets gated on evidence presence before analysis.
- Evidence-gated LLM judgment using criterion-specific prompts with strict JSON output contracts requiring resolvable evidence citations.
- Fail-confidence validation downgrading fail signals based on direct vs indirect evidence precedence to improve precision.
- Auditable per-criterion artifact export containing the runtime evidence, decision process, prompt, and model output to enable human verification.
Datasets
- Flow-A11y dynamic WCAG benchmark — 19 scenarios / 45 criteria / 765 scored rows — curated from 19 real public websites with manual oracle labels — publicly released
Baselines vs proposed
- Generic browser-agent audit baseline: exact 5-way status accuracy = 3.3% vs FLOW-A11Y = 45.6%
- Fail precision baseline = 23.5% vs FLOW-A11Y calibrated = 41.4%
- Invalid evidence references baseline = 35 vs FLOW-A11Y = 0
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.03100.
Fig 1: illustrates the problem with a concrete scenario
Fig 2: illustrates the full analysis pipeline. In the stage
Fig 3: Pre-LLM evidence layer: unsupported criteria are resolved before
Limitations
- The benchmark covers 19 scenarios and 45 criteria but does not evaluate the full WCAG 2.2 spectrum or complicated account-based flows.
- Scenarios are manually scripted and must be carefully designed to generate evidence triggering each criterion; missing scenario coverage limits recalls.
- The method currently depends on LLM judgment quality and prompt design; errors or hallucinations may still occur despite evidence constraints.
- Runtime traces are costly to generate, requiring real browser execution and instrumentation, affecting scalability for large-scale scanning.
- Some criteria rely on heuristic or indirect evidence signals limiting fail-confidence validation precision and necessitating warning tiers.
- Adversarial or obfuscated behaviors that dynamically hide failures could challenge the approach but were not examined.
Open questions / follow-ons
- How can scenario generation be automated or scaled to maximize coverage across diverse dynamic WCAG criteria?
- What improvements to LLM prompting or alternate reasoning models could reduce hallucinations or improve granularity of accessibility judgments?
- How would FLOW-A11Y perform under adversarial conditions or dynamic obfuscation aimed at hiding accessibility failures?
- Can runtime traces and gating selectively guide remediation or in-situ repair of dynamic accessibility issues?
Why it matters for bot defense
For bot-defense and CAPTCHA practitioners, FLOW-A11Y's approach highlights the importance of interaction-aware, runtime state validation rather than static snapshot inspection. Many accessibility criteria relevant to keyboard navigation, focus management, and dynamic status announcements require evidence obtained only during actual user workflows involving multi-step interactions, asynchronous updates, or delayed UI feedback. Traditional scanners that inspect static page dumps risk missing these faults and yielding false positives or false negatives. The methodology of tracing runtime behavior, projecting evidence focused on specific criteria, gating analysis on evidence presence, and using constrained LLM judgments grounded in concrete runtime signals offers a rigorous framework to detect subtle dynamic flaws. Practitioners building bot defenses and CAPTCHAs that aim for accessible interaction flows can adapt these principles to verify that assistive technologies receive correct live updates and focus handling under real user actions rather than just initial page loads. Their testing can benefit from scenario-aware automation that exercises sequences, logs multimodal runtime evidence, and analyzes outputs with transparent evidence grounding to ensure usability for users relying on keyboard, screen readers, and other assistive technologies.
Cite
@article{arxiv2607_03100,
title={ Flow-A11y: Flow-Aware Accessibility Testing },
author={ Nasr Eddine Fliti and Leisan Kokorina and Florian Tambon and Michael Papadakis },
journal={arXiv preprint arXiv:2607.03100},
year={ 2026 },
url={https://arxiv.org/abs/2607.03100}
}