Rehearsed Multi-Agent Live Product Demonstrations with Real-Time Voice Question Answering

Source: arXiv:2606.30294 · Published 2026-06-29 · By Rahul Khedar, Mayank Malhotra, Avinash Karn, Mouli V, Prakhar Mehrotra

TL;DR

This paper addresses the expensive and complex task of creating live product demonstrations for software applications, which traditionally require expert human presenters to manually select features, perform UI interactions in a running application, narrate coherently, and answer real-time questions. Prior automation efforts focus either on instruction-driven browser agents that act on tasks without narrative or on offline narrated demo videos that are static and break under UI changes. The authors present RHETOR, a multi-agent system that integrates UI exploration and static source-code analysis into a cross-modal feature representation for scripting, rehearsing, and live delivery of rehearsed, narrated demos with real-time voice question answering. The rehearsal loop iteratively repairs failing UI locators detected on a headless browser to produce a validated script serving a synchronized live presentation with guaranteed audio-action alignment. The real-time Q&A module grounds questions in a knowledge base extracted jointly from UI and code. The system was demonstrated on four deployed web applications (including Excalidraw) and achieved high script action verification rates (σ-bar ≈ 0.92 to 1.00) after two rehearsal iterations. A detailed benchmark protocol with ten metrics across six application categories is proposed to evaluate live demo quality comprehensively beyond anecdotal case studies.

Key findings

• The rehearser’s internal locator-firing rate (σ-bar) spans 0.31–1.00 over 147 scripted actions across six pipeline sessions and four applications.
• On a substantial workload of 53 actions with full feature tier differentiation, σ-bar ≈ 0.92, rising to 1.00 after two iterations of locator repair on the public-domain app Excalidraw.
• The rehearsal repair loop converges within a maximum of 3 iterations, enforcing a convergence threshold τ=0.95.
• Actions that fail locator verification after rehearsal are gracefully degraded to narration-only segments, ensuring full demo completion with a degraded-readiness threshold τ_ready=0.7.
• The runtime enforces a strict synchronization invariant aligning the timing of browser actions to the audio narration segment-end events, preventing word-offset drift during text-to-speech playback.
• Real-time voice Q&A leverages a combined knowledge document from UI exploration and code analysis, supporting questions grounded in user-facing behavior and architectural detail.
• Latency benchmarks show multi-model LLM client calls for narration, code analysis, and QA have mean latencies ranging from 1.7 to 7.9 seconds depending on task and model.
• A comprehensive benchmark protocol is defined with 10 metrics (e.g., crawl coverage, code extraction recall, feature F1, action success, degradation rate, sync drift, QA accuracy) applied to a corpus of 10 apps across 6 categories.

Threat model

The adversary is implicitly the evolving or drifting UI and application interface that can cause automated locators to fail, undermining demo correctness. The system assumes honest access to the running app and source repository. It cannot defend against malicious attacks on the demo environment or adversarial manipulations of the application to subvert demo automation. Failures are managed via a controlled offline rehearsal phase rather than at runtime.

Methodology — deep read

The paper proposes RHETOR, a five-phase multi-agent pipeline designed to automate live product demonstrations with synchronized narration and real-time voice Q&A grounded in both UI and code context.

1. Threat model & assumptions: The system assumes a benign environment controlling the demo’s browser instance and source code repository. The adversary is implicitly any drift or inconsistency in UI elements or code that could cause locators to fail, but not a malicious attacker. Runtime correctness is assured by rehearsing and repairing locator failures offline.

2. Data: Input comprises a live running web application (URL) and its source-code repository. The paper evaluates on 4 deployed web applications including public-domain Excalidraw. A benchmark corpus is defined with 10 open-source apps stratified across 6 categories (CRUD, dashboard, CMS, communications, design, data platform). Application exploration is bounded (max 30 pages, depth 4). Code reading fetches repos via GitHub API or shallow clone, batching files by directory and size (~60 KB batches).

3. Architecture / algorithm:

• Phase 1: Concurrent UI exploration via breadth-first crawl building a navigation graph G with pages up to depth 4, extracting structured page metadata and interactive elements, supplemented by a multimodal LLM summarizing screenshots. Simultaneously, static code analysis produces route manifests, data models, framework label, and architecture summaries.
• Phase 2: A single LLM call merges G and C into a feature set F with discrete focus tiers {HERO, SUPP, MENTION} and continuous demo priority ρ ∈ [1,10]. It also generates a Markdown knowledge document K partitioned for efficient query retrieval.
• Phase 3: Grounded scripting forces all actions to target UI elements discovered during exploration, dispatched by an ordered locator tuple prioritizing semantic properties (role+name, text, label) over brittle CSS selectors. Actions include navigate, click, fill (with realistic demo data via small LLM), hover, scroll, wait, highlight.
• Phase 4: Rehearsal runs the scripted demo on a real headless Chromium browser, verifying if locators resolve and actions succeed. Failed locators trigger an LLM-based repair that proposes alternative locator strategies prepended to the list. Iterations continue up to a max of 3 or until a 0.95 success threshold is met. Residual failed actions degrade to narration-only segments.
• Phase 5: Live presentation serves the rehearsed demo via a reverse proxy embedding the target app at same origin and dispatches validated locators through the client DOM. A segment synchronization invariant ties action dispatch timing precisely to the end of the textual narration audio playback using a server–client handshake.

The real-time voice QA module bridges browser microphone audio to a server-side voice activity detection and speech-to-speech WebSocket endpoint, injecting the knowledge document slice relevant to the query to ground responses in UI and code features.

4. Training regime: The system is a rule-and-LLM hybrid rather than learning-based; no training epochs or parameter optimization applies but the LLMs invoked use known API endpoints for structured calls with temperature 0.

5. Evaluation protocol: The authors define a detailed benchmark protocol with 10 metrics measuring crawl coverage, code extraction recall, feature extraction precision/recall (against human annotation), scripted action success per iteration, repair effectiveness, degradation rate, live runtime latencies, synchronization drift, QA turn latency, and QA grounding accuracy judged by blind expert review. They distinguish 6 app categories and propose ablations for rehearsal vs no rehearsal and cross-modal vs UI-only signals.

6. Reproducibility: The system implementation is ~6000 lines of Python, based on Flask and Playwright. Code runs over an OpenAI-compatible LLM client with adapters for multi-provider routing and supports streaming modalities for chat, TTS, and realtime speech. A unit test suite with 129 tests covers core components. No public code or dataset release is stated. The benchmark protocol is specified for open reproducibility.

Technical innovations

• Cross-modal feature representation merging UI exploration with source-code repository analysis into features tagged by discrete focus tiers and demo priority scores.
• A grounded scripter constrained to dispatch browser actions only on UI elements observed during exploration, using a prioritized tuple of semantic locator strategies to maximize robustness against UI drift.
• A novel rehearse-then-present pipeline with an explicit iterative loop that repairs failing locators via LLM-generated alternatives, converging within a fixed iteration cap and gracefully degrading failed actions to narration-only rather than removal.
• A runtime synchronization invariant enforcing that each browser action dispatch is precisely aligned to the audio-end event of the corresponding narration segment via a server-client handshake, eliminating text-to-speech word-offset drift.
• A real-time voice question answering path bridging browser microphone audio to server-level voice activity detection and speech-to-speech endpoints, grounded in a dynamically retrieved knowledge document constructed jointly from UI and code.

Datasets

• Excalidraw — open-source public domain whiteboard application — public GitHub repository
• Cal.com, Ghost, Plausible, Supabase Studio, NocoDB, Directus, Mattermost, Penpot, Appwrite — total 10 applications spanning CRUD, dashboard, CMS, communications, design, data platform categories — sourced from public and gated GitHub repositories

Baselines vs proposed

• Unrehearsed script (no rehearsal): mean Succ@1 action success significantly lower than rehearsed script; rehearsal raises action success rates across all application categories (exact numbers not reported).
• Cross-modal merge vs UI-only: ablation planned to measure feature extraction precision and recall, and code-grounded Q&A accuracy; hypothesis is that removing code analysis collapses code-grounded accuracy (results pending).

Limitations

• Evaluation presented is preliminary with only four deployed applications and 147 scripted actions; the comprehensive benchmark protocol is defined but not fully executed yet.
• No reported adversarial evaluation or robustness tests against deliberate UI manipulation or malicious code changes.
• The system relies on access to source-code repositories for code analysis; closed-source or incomplete repos limit applicability.
• Rehearsal iteration capped at three limits recovery from highly dynamic or deeply unstable UI changes.
• Recorded demo packages lose the interactive Q&A capability, reducing their utility compared to live presentations.
• The system supports only web applications with crawling and access to running instances; native or non-web app demos are out of scope.

Open questions / follow-ons

• How does rehearsal performance and convergence scale with larger applications and more complex interaction graphs beyond tested examples?
• Can locator repair LLM prompts be generalized or automated to handle adversarial UI changes or deliberate locator obfuscation?
• What is the user experience impact of degraded narration-only segments in live demos, and can fallback strategies be improved?
• How effective is the real-time question answering module under noisy environments, multilingual usage, or adversarial questioning?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, RHETOR’s approach to integrating multi-modal static and dynamic context for automated UI interaction and narration scripting highlights the importance of grounding automation in both runtime interface state and source code. This level of multi-modal context could inspire more robust bot-detection mechanisms that verify consistency at both the UI and backend code model levels rather than relying solely on behavioral signals. The rehearse-then-present rehearsal loop that iteratively repairs locator failures can parallel techniques in bot defense where multiple passes over observed user events refine anomaly detection models. Additionally, the synchronization mechanism eliminating text-to-speech drift through server-client handshakes could be informative for coordinating multi-modal signals in anti-bot verification challenges involving audio-visual modalities. The real-time voice Q&A system grounded in application knowledge also opens avenues for conversational bot challenges that test deep contextual understanding beyond static CAPTCHAs.

Cite

@article{arxiv2606_30294,
  title={ Rehearsed Multi-Agent Live Product Demonstrations with Real-Time Voice Question Answering },
  author={ Rahul Khedar and Mayank Malhotra and Avinash Karn and Mouli V and Prakhar Mehrotra },
  journal={arXiv preprint arXiv:2606.30294},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.30294}
}

Read the full paper

• arXiv:2606.30294 (abstract)
• arXiv:2606.30294 (PDF)