From Forecasts to Auditable Reports: Evidence Contracts for LLM-Assisted Housing-Guarantee Risk Monitoring

Source: arXiv:2607.14026 · Published 2026-07-15 · By Hyeongcheol Kim, Yoontae Hwang

TL;DR

This paper addresses the challenge of converting next-month housing-guarantee risk forecasts into auditable, operationally useful reports. The domain is the South Korean jeonse deposit guarantee market, where upper-tail risk events are sparse, raw data is confidential, and naive narrative generation by language models risks fabricating unsupported claims. The authors propose an end-to-end evidence-constrained pipeline that links an interpretable risk-sensitive Temporal Fusion Transformer (TFT) forecasting backbone with a retrieval mechanism that finds historical precedent windows matching the forecaster’s rationale using Centered Kernel Alignment (CKA). Structured evidence contracts expose admissible facts (forecast, drivers, lookback periods, analogous cases, risk references) to an LLM-based report generator constrained to produce verifiable narratives. A hard-rule audit verifies numerical, temporal, and claim fidelity before human analyst review.

Empirically, the tailored regret-sensitive TFT backbone improves tail-risk detection recall from 16% to 56% at the P90 level compared to vanilla MAE-trained models, sacrificing only 1.4%p in average prediction error. Across eight LLMs, incorporating typed evidence contracts consistently enhances report numerical fidelity, claim grounding, and overall quality versus raw-data prompts. A practitioner evaluation with 51 analysts endorses the practical utility of the approach, recognizing that evidence-anchored generation combined with automated audits plus expert oversight produces reliable, actionable risk monitoring reports. This work demonstrates that achieving accountable, high-stakes monitoring with LLM assistance requires coupling forecasting, retrieval, constrained generation, and auditing into a unified workflow rather than an end-to-end generative approach.

Key findings

• The regret-sensitive TFT model achieves a P90 recall of 0.56 on 246 test observations for next-month guarantee accident risk, markedly higher than 0.16 for a vanilla MAE-trained TFT and 0.08 for LSTM-MAE baseline.
• This improved tail recall comes at a cost of increasing MAE from 0.027 (LSTM-MAE) to 0.041 (TFT-regret), a 1.4%p increase but operationally justified by catching more high-risk events.
• Using structured evidence contracts improves mean report quality scores across eight LLMs from an average of 7.97 (raw data input) to 8.25–8.77 (evidence card or structured draft interfaces) on a 0–10 scale.
• Hard rule audits detect unsupported inference, unit mistakes, and direction-label inconsistencies, preventing LLM hallucination and ensuring factual fidelity before analyst review.
• Retrieval via CKA similarity on model rationale matrices identifies admissible historical windows with comparable variable-time emphasis rather than raw covariate similarity, enabling analogy without forecast leakage.
• The evidence contract enforces that LLM-generated reports cannot introduce new forecasts or claim unsupported historical analogues, bounding generation to verifiable facts.
• Practitioner evaluation involving 51 analysts rates the resulting reports as practically useful and supports operational pilots integrating LLM assistance with human oversight.
• The pipeline’s explicit separation between forecasting, retrieval, evidence contract formation, constrained LLM generation, and auditing addresses governance and data confidentiality requirements inherent in financial risk monitoring.

Threat model

The adversary is essentially the untrusted language model component, which can hallucinate unsupported information, distort numerical values, fabricate historical analogues, or violate data governance by exposing confidential raw data. The system assumes an honest-but-curious LLM restricted by input constraints and explicit evidence contracts to prevent unverified claims. Human analysts act as the ultimate check, so adversaries cannot fully automate false reports or bypass oversight.

Methodology — deep read

1. Threat model & assumptions: The adversary is any untrusted automated generation system (the LLM) that might hallucinate numerical, temporal, or causal claims unsupported by evidence. Raw confidential guarantee records cannot be directly exposed. The system assumes human analysts ultimately review audited reports before operational decisions. The LLM’s role is limited to verbalizing pre-approved evidence to prevent fabrication.

2. Data: Monthly panel data on South Korean jeonse housing guarantee completions from September 2015 to December 2025, covering 132 region-by-housing-type segments and 124 months. After filtering for completeness and continuity, 82 usable series produce 4,580 windows, with 246 held-out test observations. Labels are next-month guarantee accident rates (accidents/completions). For report generation experiments, synthetic aggregate explanation data calibrated to empirical ranges is created to avoid direct exposure of confidential source data.

3. Architecture/algorithm: The forecasting backbone is a Temporal Fusion Transformer (TFT) [26], which produces next-month scalar forecasts alongside variable-selection and temporal-attention weights. These weights form a rationale matrix representing the model’s reasoning over L lookback months and m inputs. A regret-sensitive custom loss prioritizes detecting right-tail risk by imposing higher penalty for under-prediction than over-prediction. Historical window retrieval compares rationale matrices via Centered Kernel Alignment (CKA) to find admissible analogous cases with similar model rationales, not just raw feature proximity.

Evidence contracts are typed data structures storing forecast, recent baseline, direction label, key drivers, timing evidence, retrieval analogues, high-risk reference windows, and sample support info. These form the fixed input to LLM generation. The LLM produces textual reports referencing only admissible evidence fields. The evidence contract thus acts as a binding constraint.

4. Training regime: Models are trained on contiguous windows with complete covariates. TFT's regret-sensitive loss weighting parameter (lambda) controls tradeoff between average error and tail recall. Specific hyperparameters, batch sizes, optimizer details, training epochs, or seeds are not fully detailed but training aligns with the temporal fusion transformer approach. Retrieval and evidence production happens only after forecasts are fixed.

5. Evaluation protocol: Forecasting is assessed on 246 held-out windows for MAE, RMSE, and recall at upper 30% (P70) and 10% (P90) tail risks, plus asymmetric regret penalties. Report generation is evaluated on fixed 24 synthetic aggregate cases varying direction and evidence availability. Eight LLMs generate responses for four tasks (Q1–Q4) testing driver and timing evidence extraction, direction label fidelity, and structured report generation. Three evidence-defined interfaces (Raw data, Evidence card, Structured draft) isolate the effect of structured evidence and report grammar. Report fidelity is scored on a 0–10 scale, combined with rule-based hard audits detecting unsupported claims, label errors, numerical mistakes, and logical inconsistencies. Practitioner evaluations with 51 analysts assess practical usefulness.

6. Reproducibility: Code and datasets are not publicly released due to confidentiality of guarantee records. Synthetic reporting scenarios are documented in the appendix to enable partial reproduction of the evidence interface and reporting experiments. Forecasting models are standard TFT implementations with a modified asymmetric loss. Retrieval relies on explicit CKA metric described in eqn 4. The audit protocol is rule-based and deterministically defined.

Example end-to-end: For a target segment and month, the TFT produces a forecast risk value, variable importance, and temporal attention matrices. Using the rationale matrix, historical windows are retrieved by maximizing CKA similarity among admissible past windows. Along with the forecast, baseline, and direction label, these build the evidence contract, fixing admissible facts and analogues. This contract is fed, along with raw and structured data, into an LLM prompt constrained to verbalize only vetted evidence fields. The draft report undergoes an automated audit for numerical, directional, unit, and reasoning fidelity. Flags are highlighted for analyst review, who can accept, revise, or reject before operational deployment.

Technical innovations

• Use of regret-sensitive loss with TFT to optimize tail-risk recall in forecasting while preserving interpretability signals for retrieval.
• Centered Kernel Alignment (CKA) similarity applied to combined variable-selection and temporal-attention rationale matrices to retrieve historical analogues aligned with model reasoning instead of raw features.
• Definition of typed evidence contracts that strictly constrain LLM generation to pre-approved, admissible facts (forecasts, drivers, temporal windows, analogues) eliminating hallucination.
• Integration of automated hard-rule audits verifying numerical, temporal, directional, and claim fidelity in LLM-generated reports before analyst review.
• Structured report grammar interface that enforces a fixed evidence verbalization order, reducing reviewer burden and improving report grounding.

Datasets

• South Korean jeonse deposit guarantee panel — ~82 series × 124 months = 4,580 windows — confidential, not publicly released
• Synthetic aggregate explanation scenarios — 24 cases calibrated to empirical data ranges — constructed for LLM reporting evaluation

Baselines vs proposed

• LSTM (MAE loss): MAE=0.027, P90 recall=0.08 vs TFT (regret loss): MAE=0.041, P90 recall=0.56
• DLinear (MAE loss): MAE=0.039, P90 recall=0.44 vs TFT (regret loss): MAE=0.041, P90 recall=0.56
• LightGBM (MAE loss): MAE=0.036, P90 recall=0.36 vs TFT (regret loss): MAE=0.041, P90 recall=0.56
• Raw Data interface report quality avg score (various LLMs) ≈ 7.97 vs Evidence Card interface ≈ 8.25–8.77

Limitations

• No public release of confidential housing guarantee panel data limits independent verification and reproduction.
• Forecasting evaluation emphasizes recall of extreme upper-tail events at the expense of some increase in average prediction error, which may not suit all risk management contexts.
• The retrieval method yields analogous historical windows based on representational similarity, which are not causal matches and might mislead if interpreted incorrectly.
• Experiments evaluate structured reporting only on synthetic aggregate scenarios; performance on fully real operational data remains to be validated.
• The evidence contract limits generation but may reduce report fluency or nuance; user acceptance beyond domain analysts is untested.
• Human analyst oversight remains crucial; the system does not automate final risk decisions or replace expert judgment.

Open questions / follow-ons

• Can the evidence contract framework be extended to dynamically incorporate new data sources or evolving macro-financial indicators in real time?
• How robust is CKA-based rationale retrieval under distribution shifts or market regime changes not seen in training?
• What are the tradeoffs between tighter generation constraints for auditability versus richer, more interpretable narrative explanations acceptable to domain experts?
• Could adversarial attacks on the retrieval mechanism or evidence contract inputs induce misleading analogues or override audit safeguards?

Why it matters for bot defense

For bot-defense and CAPTCHA engineers, this paper illustrates a rigorous methodology for constraining large language model outputs with typed evidence contracts combined with automated audits to maintain factual fidelity in high-stakes domains. The approach addresses known risks of unmoored LLM hallucinations by forcing alignment to verifiable model outputs and retrieved analogous evidence instances. Practitioners in bot defense could translate this concept to constraining LLM-generated challenge-solving justifications or decision explanations, ensuring that automated assessments are tied to audit-ready, structured evidence rather than free-form generation. The layered architecture—combining interpretable forecasting, principled retrieval in model reasoning space, constrained generation, and deterministic audit rules—offers a replicable pipeline blueprint for safely incorporating LLMs in security-sensitive automation tasks where evidence provenance and governance matter.

Cite

@article{arxiv2607_14026,
  title={ From Forecasts to Auditable Reports: Evidence Contracts for LLM-Assisted Housing-Guarantee Risk Monitoring },
  author={ Hyeongcheol Kim and Yoontae Hwang },
  journal={arXiv preprint arXiv:2607.14026},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.14026}
}

Read the full paper

• arXiv:2607.14026 (abstract)
• arXiv:2607.14026 (PDF)