LLM-Based Examination of Eligibility Criteria from Securities Prospectuses at the German Central Bank

Source: arXiv:2606.27316 · Published 2026-06-25 · By Serhii Hamotskyi, Akash Kumar Gautam, Christian Hänig

TL;DR

This paper addresses the challenge of automating the verification of securities' eligibility as collateral at the German Central Bank, a process which traditionally requires manual review of lengthy, bilingual, semi-structured prospectuses. Prior approaches based on Named Entity Recognition (NER) imposed constraints such as sensitivity to OCR artifacts, linguistic variance, rigid span boundaries, and heavy annotation requirements. The authors present the first case study applying generative Large Language Models (LLMs) to this task, replacing token-classification with a pipeline that decomposes eligibility verification into extraction, normalization, and interpretation steps. This methodology provides greater flexibility handling noisy text and mixing of German-English content.

They also introduce a novel value-based evaluation using LLM-as-a-judge to semantically assess extraction quality rather than rigid location-based metrics. Experiments on a corpus of 413 prospectuses show that LLM-based systems like Llama 3.3 70B and Cohere Command-R 08-2024 achieve high per-document eligibility precision up to 91%, with a conservative bias to minimize false acceptance of ineligible securities. The approach significantly outperforms a previous transformer-based NER system on complex criteria, demonstrating the practical and methodological advancements of generative extraction and semantic evaluation in a challenging financial NLP domain.

Key findings

• The proposed LLM-based pipeline reached up to 91% precision for document-level eligibility classification, outperforming prior NER-based methods by approximately 20 percentage points in precision.
• Cohere Command-R 08-2024 (32B parameters) slightly outperformed the larger Llama 3.3 70B-Instruct model across most metrics despite being half the size.
• Value-based evaluation with LLM-as-a-judge provided more semantic and robust assessment compared to traditional offset-based or fuzzy string matching techniques, especially handling bilingual and OCR noisy data.
• Among criteria, currency and principal amount were easiest to extract accurately (>98% accuracy), while ‘‘redemption at maturity’’ and complex criteria like coupon and status were harder, but still improved by LLM approaches.
• The system adopts a safety-biased conservative decision logic, resulting in high false negative rates but minimizing false positives to reduce financial risk from accepting ineligible collateral.
• The dataset comprised 413 prospectuses with about 29% labeled as ineligible; bilingual (German-English) documents accounted for about a third of samples, posing additional linguistic challenges.
• The pipeline reduces dependence on expensive manual annotation by leveraging instruction-following zero-shot LLM capabilities, enabling easier adaptation to new or evolving eligibility criteria.
• LLM-based extraction and interpretation separated into multi-stage pipeline mitigated hallucination risks by restricting each step’s input context and enabled provision of semantic justification for decisions.

Threat model

The adversary is implicitly the financial risk that the central bank faces from mistakenly accepting ineligible securities as collateral; the system assumes no attacker beyond noise in OCR and bilingual, non-standard document layouts. Adversaries cannot modify the models or annotations directly but may produce noisy or ambiguous prospectus text. The model must balance precision to avoid false acceptance while allowing manual review for uncertain cases.

Methodology — deep read

The study begins with the threat model of automating collateral eligibility verification at a central bank, where the adversary is absent but the domain demands high precision to avoid financial risk. The system assumes access to prospectus PDFs (English/German bilingual, semi-structured, with OCR noise) and some master data metadata; the models cannot fully rely on extensive annotated data due to annotation cost and variability.

The dataset contains 413 prospectuses split into 268 training and 145 test documents, with 18 manually annotated entity types relevant to eligibility criteria. Approximately one-third of documents are bilingual with parallel or interleaved German-English text. Annotation includes marking only evidential text spans, ignoring irrelevant mentions. The test set was doubly annotated for reliability.

The pipeline architecture uses Large Language Models (LLMs) for three sequential generative steps: (1) Extraction—to detect eligibility evidence and produce normalized structured JSON values including raw text and context quotes; (2) Normalization—standardizing varied textual inputs (currencies, amounts); (3) Interpretation—mapping normalized extracted values via decision rules and master data to produce binary eligibility predictions on six criteria (four simple, two complex).

PDFs are first converted to Markdown text (via Docling) preserving structure and reducing OCR artifacts before input to LLMs. Two LLMs were tested: Llama-3.3-70B-Instruct (baseline with strong reasoning) and Cohere Command-R 08-2024 (32B, multilingual and trained for grounded generation). LangChain was used to prompt models for structured JSON output with low temperature (0.1). Frequency penalties and text normalization addressed output formatting errors.

Evaluation is on three levels: document-level eligibility binary classification, per-criteria binary classification, and value extraction accuracy. Unlike prior span-offset metrics, they adopted a semantic value-based evaluation comparing extracted values to multiple ground truths using fuzzy string matching and an LLM-as-a-judge (Mistral-Small-3.1-24B-Instruct) to score equivalence semantically. This approach handles bilinguality, OCR noise, and paraphrases.

Comparisons are made to the previous transformer-based NER classifier from Hänig et al. (2023). The LLM pipeline requires no supervised fine-tuning, removing manual annotation bottlenecks. Inference requires multiple LLM calls per document (2 for simple, 1 per complex criterion), incurring higher latency than NER but gaining flexibility.

A concrete end-to-end example: the LLM extracts "principal_amount" as {"raw_value": "up to 10.000,00€", "value": "up to EUR 10.000", "evidence": {...}}; then it interprets this normalized value to determine eligibility based on fixed or conditional amounts. The final eligibility prediction across all six criteria is assembled programmatically. Evaluation metrics (precision, recall, F1) are computed on these predictions using value-based matching accounting for multiple annotations per field.

Reproducibility is partial; code is not publicly released but model weights are standard open LLMs. Dataset is private but derived from the published FinCorpus-DE10k corpus. Evaluation relies on LLM-as-a-judge with human-written equivalence rules for explainability. No adversarial evaluation or distribution shift experiments are reported.

Overall, the methodology uses a robust generative extraction pipeline leveraging large pre-trained models and a semantic-evaluation paradigm better suited to noisy, bilingual, complex financial documents than traditional NER plus span metrics.

Technical innovations

• Introduction of a generative, multi-stage LLM pipeline for extracting and interpreting eligibility criteria from semi-structured, bilingual financial prospectuses, moving beyond rigid token classification.
• Development of a value-based evaluation methodology using an LLM-as-a-judge semantic comparison of extracted evidence to multiple ground truths, more robust to linguistic variance and OCR noise than standard offset-matching.
• Integration of normalization and interpretation as distinct LLM steps, separating data extraction from eligibility decision logic to reduce hallucination and improve explainability.
• Use of multilingual instruction-following LLMs (e.g. Cohere Command-R 08-2024) with very large 128k token context windows to process entire prospectuses directly without chunking.

Datasets

• FinCorpus-DE10k — 413 prospectuses annotated for eligibility evidence — private proprietary dataset derived from German financial documents

Baselines vs proposed

• Hänig et al. (2023) NER approach: per-document eligibility precision = ~70% vs LLM Command-R 08-2024: precision = 91%
• Hänig et al. (2023) NER approach: F1 on complex criteria (coupon, status) < 0.93 vs LLM pipeline cohort: coupon F1 = 0.97, status F1 = 0.94
• Hänig et al. (2023) offset-based extraction F1 = 0.72 vs LLM-based value extraction evaluated by LLM-as-a-judge F1 = 0.85–0.86
• Llama-3.3-70B-Instruct accuracy = 0.82 vs Command-R 08-2024 accuracy = 0.84 on document-level eligibility

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.27316.

Fig 1: Sample annotated paragraphs. Note the varying length and complexity of different types, as

Fig 3: F1-scores of extracted values. Hänig

Limitations

• The system incurs lower recall due to cascading false negative logic, favoring safety at the expense of missing some eligible securities.
• Evaluation lacks large-scale manual adjudication of LLM-as-a-judge reliability; the judge itself may introduce unknown biases.
• The dataset, while sizable, is proprietary and limited to German and some bilingual documents; generalizability to other languages or financial domains is unknown.
• The approach relies on accurate text extraction from PDFs via Markdown conversion, which struggles with tables and complex layouts; no vision-based document parsing is currently integrated.
• Inference latency is high (tens of seconds per document) compared to NER methods, limiting real-time throughput.
• No adversarial testing or robustness to deliberate evasion by maliciously crafted prospectuses was conducted.

Open questions / follow-ons

• How can vision-language or OCR-free document understanding models be integrated to better handle complex PDF layouts, tables, and checkboxes?
• What methods can provide more reliable, explainable, and unbiased meta-evaluation for LLM-as-a-judge frameworks to ensure long-term assessment validity?
• Can Retrieval-Augmented Generation (RAG) approaches that ground LLM outputs in specific source document spans improve hallucination mitigation, efficiency, and explainability in this domain?
• How does the system perform under distribution shifts over time, such as changes in legal language or financial instruments, and how can continual adaptation be supported?

Why it matters for bot defense

This work demonstrates the potential and challenges of applying large, instruction-tuned LLMs in complex, safety-critical document information extraction for financial regulatory tasks. For bot-defense and CAPTCHA engineers, the paper highlights key techniques for leveraging generative LLM pipelines to handle noisy, multilingual text with minimal annotated data, an increasingly common scenario when dealing with web-scale or user-generated content verification.

The multi-stage decomposition (extraction, normalization, interpretation) combined with a flexible semantic evaluation (LLM-as-a-judge) offers a blueprint for other domains requiring high precision under uncertainty. However, the trade-offs in latency and the need for robust document parsing suggest that pre-filtering or hybrid pipelines integrating discriminative and generative approaches—and potentially grounding results for explainability—are important considerations. Understanding how to design and evaluate extraction systems resistant to input noise and representation variability is critical knowledge transferable across bot detection and challenge-response systems relying on document or text analysis.

Cite

@article{arxiv2606_27316,
  title={ LLM-Based Examination of Eligibility Criteria from Securities Prospectuses at the German Central Bank },
  author={ Serhii Hamotskyi and Akash Kumar Gautam and Christian Hänig },
  journal={arXiv preprint arXiv:2606.27316},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.27316}
}

Read the full paper

• arXiv:2606.27316 (abstract)
• arXiv:2606.27316 (PDF)