Evidence-in-the-Loop: Trace-Driven Optimization for Customer-Service LLM Agents

Source: arXiv:2607.18039 · Published 2026-07-20 · By Chunming Wu, Dafei Qiu, Congde Yuan, Charles Quan, Jun Wu, Suipeng Li et al.

TL;DR

This paper addresses the challenge of deploying large language models (LLMs) for customer-service bots with strict evidence grounding, compliance to business policies, and human handoff safeguards. The authors present an Evidence-Grounded Customer-Service Agent Workflow that integrates multi-channel FAQ retrieval (lexical BM25 and semantic vector), weighted Reciprocal Rank Fusion (RRF), and cross-encoder reranking to produce auditable FAQ candidates. These candidates are combined with scenario-specific rule-derived evidence, conversation memory, and clarification state within a fixed LangGraph DAG for controlled final LLM issue/action selection. The workflow enables fine-grained trace-driven pipeline diagnosis to identify failures in recall, ranking, evidence fusion, clarification, or action policy, guiding targeted improvements. This is demonstrated in a real-world deployment with 605 anonymized sessions and detailed metrics.

Key results show that hybrid multi-channel recall achieves 96.76% top-50 coverage and reranker fine-tuning via teacher-score distillation improves Hit@1 on held-out data from 56.76% to 75.68%. On a sealed multi-turn session test, reranker improvements raise issue-turn Rerank@1 from 69.4% to 79.2%, and final LLM decision post-training via direct preference optimization boosts session-level grounded accuracy from 88.5% to 90.5%. Post-launch operational review confirms the full workflow improves accuracy from 79% (legacy RAG-only) to 89.5%. The paper emphasizes diagnosing and updating specific pipeline components rather than relying solely on larger backbone LLMs. The evaluation unpacks retrieval, reranking, and final decision impacts rather than collapsing into a single metric, reflecting industrial deployment complexity.

Key findings

• Hybrid recall combining BM25, issue-title embeddings, and description embeddings reaches 96.76% Hit@50 on a 309-query diagnostic subset (E1).
• The base BGE reranker has a low top-1 accuracy of approximately 56.31% on business FAQ candidates (E1 and E2).
• Teacher-score distillation fine-tunes the reranker, increasing held-out reranker Hit@1 from 56.76% to 75.68% (E2).
• Reranker improvements raised Rerank@1 at issue-turn level from 69.4% to 79.2% and KB-grounded session accuracy from 86.5% to 88.5% on a sealed 200-session test (E3).
• A direct preference optimization (DPO) trained final-decision LLM raised session accuracy to 90.5%, isolating decision-stage improvement from reranker changes (E4).
• Replacing the backbone from Qwen3.5-27B to GPT-4o changed final issue-selection accuracy by less than 1 percentage point under the diagnostic pipeline (E1), indicating backbone scale is not the dominant bottleneck.
• Post-launch evaluation shows the full Evidence-Grounded Customer-Service Agent workflow achieves 89.52% reviewed accuracy versus 79.00% for a legacy RAG-only baseline on session accuracy.
• Trace-driven failure localization enables targeted updates to KB, reranker hard negatives, decision preferences, or rules rather than blind model scaling.

Threat model

The adversary is an end user interacting with a customer-service chatbot who may attempt to elicit hallucinated, incorrect, or policy-violating responses from the LLM. The system assumes that backend rule-based evidence and retrieval modules cannot be bypassed by the user. The LLM is constrained to select from grounded evidence or trigger clarification / human handoff actions enforced by deterministic guards. The adversary cannot directly manipulate evidence sources or the model weights at runtime. Ensuring auditability, policy compliance, and bounded action space mitigates risks of malicious or misleading LLM outputs in sensitive customer-service domains.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is an end user seeking customer service answers via chat. The system must avoid hallucinated or policy-violating LLM outputs. The LLM is not allowed free generation; it must base answers strictly on grounded FAQ evidence, rule-derived evidence, or handoff policies enforced externally. Human handoff and policy guardrails limit escalation risks. Attackers cannot bypass retrieval or rule evidence mechanisms.

2. Data: The evaluation dataset comprises 605 anonymized dialogue sessions with 1,427 turns collected from production logs. After cleaning and deduplication, a T-Set FAQ subset of 581 turns was labeled with gold FAQ issue IDs and split into 492 train, 52 dev, and 37 test queries. A diagnostic subset of 309 non-test queries was used for early pipeline diagnosis. A sealed B-Set of 200 sessions supported multi-turn review and validation. Queries are machine-translated to English. User turns are concatenated into a single retrieval query; final decision LLM input uses only the latest turn plus retrieved evidence.

3. Architecture/Algorithm: The workflow implements a LangGraph StateGraph (fixed DAG) with stages: session analysis, intent-routing LLM, dual evidence paths (hybrid RAG and rule-derived), evidence fusion, and final decision LLM for issue/action selection. The hybrid RAG evidence path performs three-channel retrieval: BM25 lexical search over FAQ titles/descriptions, KNN semantic search over issue-title embeddings, and KNN over description embeddings. Hits from these are merged using weighted Reciprocal Rank Fusion (RRF) with tuned weights and a top-50 candidate cutoff. A cross-encoder reranker (BGE base checkpoint) reranks top RRF candidates with scores computed over query-document pairs. Reranker improvements use teacher-score distillation with labeled query-FAQ positives and hard negatives mined from top retrieval.

Rule-derived evidence accesses scenario-specific context fields via a Model Context Protocol (MCP) and applies symbolic business-policy rules as predicates over structured context. Matches become typed rule-derived evidence candidates prepended to the FAQ candidates. Both evidence channels are fused for a final issue/action selection by another LLM prompt stage that examines the typed, ranked candidate list and selects answer, clarify, or human handoff.

4. Training Regime: Reranker fine-tuning used supervised teacher-score distillation on 492 training queries with positive FAQ labels and hard negatives. Specific hyperparameters, epochs, or hardware are not detailed. Final decision LLM also underwent post-training with direct preference optimization (DPO) on annotated preference pairs to improve issue/action accuracy.

5. Evaluation Protocol: Multiple evaluation blocks isolate system components—diagnostic recovery and reranker ranking on subsets (E1, E2), multi-turn session validation with full fused evidence (E3), and decision LLM preference post-training (E4). Hit@K is the primary metric at recall and rerank stages. Session accuracy measures end-to-end issue/action correctness on multi-turn sets. Human-in-the-loop annotation and review validate trace-driven error attribution (retrieval misses, reranker errors, policy mismatches, clarification faults). Comparisons include the base BGE reranker, fine-tuned reranker, Qwen3.5-27B vs GPT-4o backbones, and legacy RAG-only workflows. Post-launch operational review compares live samples.

6. Reproducibility: Code and exact models are not publicly released. Dataset provenance is proprietary anonymized real-world customer-service logs from the deployment platform. Details on seed control or exact model weights are not described. The method emphasizes integration within a production environment including LangGraph workflow and multi-layer evidential controls rather than black-box LLM experiments.

Concrete End-to-End Example: For a user query "I cannot withdraw my funds," the intent-routing LLM rewrites the query and determines required context. The hybrid retrieval fetches lexical and semantic FAQ candidates. Concurrently, MCP reads user session context to match rules (e.g., withdrawal restrictions). The rule-derived evidence and FAQ candidates are fused and reranked. The decision LLM reads the structured candidate list and selects an issue ID plus the action (answer with grounded KB info, clarify missing details, or human handoff). The response text is produced with explicit citation of evidence, bounded by policy and clarification limits. Failures from reranker misordering or missing evidence trace to targeted updates in KB or model fine-tuning.

Technical innovations

• Hybrid RAG evidence construction combining lexical BM25 recall, two semantic recall channels, weighted Reciprocal Rank Fusion, and cross-encoder reranking optimized for transparent FAQ candidate generation.
• Evidence-grounded issue/action decision module fusing typed FAQ and scenario-specific rule-derived evidence in a fixed LangGraph DAG with bounded LLM selection actions (Answer, Clarify, Human Handoff).
• Trace-driven review loop using replayed logged conversations to exactly localize failures in pipeline stages and convert human-reviewed errors into targeted KB updates, reranker hard negatives, prompt or rule fixes, and preference-tuning datasets.
• Application of teacher-score distillation to BGE reranker fine-tuning that improves top-1 FAQ issue selection accuracy significantly without changing retrieval architecture.
• Separation of routing/guardrail logic and external context with MCP to enforce strict policy and evidence boundaries, maintaining auditability and preventing free-form LLM hallucination.

Datasets

• T-Set FAQ subset — 581 labeled queries — anonymized historical production conversations
• Diagnostic subset — 309 queries — non-test portion of T-Set
• B-Set session validation — 200 sealed multi-turn sessions — proprietary real-world deployment data

Baselines vs proposed

• Base BGE reranker: Hit@1 = ~56.31% vs Distilled reranker: Hit@1 = 75.68% (E2 held-out test)
• Hybrid recall top-50 coverage: 96.76% Hit@50 (E1 diagnostic)
• Issue-turn Rerank@1: Base reranker = 69.4% vs Distilled reranker = 79.2% (E3 B-Set)
• KB-grounded session accuracy: Base reranker = 86.5% vs Distilled reranker = 88.5% (E3 B-Set)
• Final decision LLM (DPO trained): Session accuracy = 90.5% (E4 B-Set)
• Legacy RAG-only workflow session accuracy = 79.00% vs Evidence-Grounded Agent workflow = 89.52% (post-launch review)

Limitations

• No comprehensive labeling reported yet for clarification precision, rule-trigger precision, or full MCP/rule-evidence coverage.
• Reranker fine-tuning details (hyperparameters, epochs) and hardware specs are not fully disclosed, limiting exact reproducibility.
• Dataset is proprietary with no public release, limiting independent validation or comparison.
• Evaluation sets do not cover all possible operational scenarios, especially edge cases with incomplete evidence or adversarial user inputs.
• Human handoff and policy rule layers are treated as opaque deterministic systems, with limited insight into handoff errors or rule conflicts.
• Final decision LLM improvements tested on relatively small sealed session set (200 sessions) with unknown statistical power.

Open questions / follow-ons

• How to further quantify and improve clarification precision and coverage for incomplete or ambiguous user queries?
• What are the impacts on performance and safety when scaling to larger or more diverse FAQ corpora and more complex rule condition sets?
• How to integrate stronger adversarial robustness testing and detect malicious input exploitation or model bias in evidence-grounded workflows?
• Can the trace-driven improvement loop be automated further without human annotation while preserving auditability and compliance?

Why it matters for bot defense

This paper presents a practical, evidence-grounded orchestration approach for deploying LLM agents in regulated, high-stakes customer-service settings, balancing model capacity with explicit retrieval, rule-based policy, and human-in-the-loop oversight. For bot-defense and CAPTCHA practitioners, such an evidence-grounded pipeline shows that scale alone does not ensure safe/autonomous agent actions. Instead, tightly controlled evidence retrieval and fusion combined with trace-driven error attribution provide a robust framework to avoid hallucination and enforce compliance, critical for protecting against automated fraudulent behaviors or misuse. The modular LangGraph DAG design and auditable candidate contract enable systematic monitoring and targeted improvements, techniques which can be adapted to security-sensitive dialog or bot-detection systems requiring strong behavioral guarantees against adversarial input or unsafe autonomous agent responses.

Cite

@article{arxiv2607_18039,
  title={ Evidence-in-the-Loop: Trace-Driven Optimization for Customer-Service LLM Agents },
  author={ Chunming Wu and Dafei Qiu and Congde Yuan and Charles Quan and Jun Wu and Suipeng Li and Mo Wu and Gavin Xie and Hope Chen and Max Yao },
  journal={arXiv preprint arXiv:2607.18039},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.18039}
}

Read the full paper

• arXiv:2607.18039 (abstract)
• arXiv:2607.18039 (PDF)