ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction

Source: arXiv:2607.29677 · Published 2026-07-31 · By Boyang Zhang, Adrian Lyjak, Eli Stewart, Zhaoqi Li, Simon Suo

TL;DR

ExtractBench addresses the enterprise challenge of schema-guided extraction, where given a schema and a document, an AI agent extracts structured data with precise source grounding. Existing benchmarks are limited in coverage, schema flexibility, grounding measurement, or cost accounting. ExtractBench fills this gap with a comprehensive benchmark comprising 370 documents across 8 domains and 67 document types, including born-digital, scanned, and handwritten documents. Its evaluation jointly measures value accuracy, record completeness, fine-grained visual grounding, and cost per page. The benchmark uses a multi-source, scalable ground-truth pipeline combining independent system agreement, synthetic data, and human verification to ensure high-quality references. The authors evaluate 14 recent extraction systems, showing that commercial vision-language models (VLMs) perform well on short documents but falter on long or complex ones. Coding agents achieve higher accuracy but at much higher costs. Their LlamaExtract Agentic Plus achieves top accuracy (95.6% F1) comparable to coding agents but at less than half the cost, with strong performance across challenging document types and lengths. ExtractBench thus provides a practical, broad, and rigorous testbed to drive enterprise document extraction research and deployment.

Key findings

• ExtractBench covers 370 documents totaling 4,869 pages, from 8 business domains and 67 document types.
• The benchmark tags documents across 5 axes—task, perception, table structure, length, domain—enabling detailed failure analysis.
• Ground truth is generated via a hybrid pipeline: ensemble agreement for real docs, programmatic synthetic list generation, and human verification for scanned forms.
• Value accuracy is scored by order-insensitive value F1; grounding by word-level F1 (with IoU threshold 0.5) and page-level F1.
• On short documents (≤10 pages), commercial VLMs achieve up to ~88% F1, but on long documents (>50 pages) accuracy can drop below 40%.
• Coding agents maintain higher accuracy on long documents (Claude Code Opus 4.8 achieves 88.1% F1 on long docs) but at costs >15¢ per page.
• LlamaExtract Agentic Plus ranks highest overall with 95.6% F1 and costs 8.1¢ per page, outperforming coding agents both in cost and accuracy.
• Only specialized APIs and LlamaExtract variants return word-level boxes suitable for visual grounding, with LlamaExtract Agentic Plus achieving 46.4% word-level grounding F1 and 84.9% page-level grounding F1.

Threat model

The adversary corresponds to the natural challenges in enterprise documents that complicate automated extraction: heterogeneous document layouts, noisy scans, handwritten content, very long or dense tables, and ambiguous fields. The system must extract all requested values according to a user-defined schema, correctly handling nulls and repeated records without hallucination or omission. Malicious adversaries who intentionally alter documents or insert deceptive content are out of scope.

Methodology — deep read

1. Threat Model & Assumptions: ExtractBench targets enterprise document extraction where adversaries include varied document layouts, noisy scan sources (including handwriting), and complex table structures. The system assumes input documents plus a user-defined JSON schema describing the fields to extract. The challenge is faithfully extracting all requested values with correct grounding, even in the presence of noise, ambiguity, and very long or dense documents. Adversaries in this sense are not malicious users but structural and perceptual challenges causing extraction failures.

2. Data: The dataset contains 370 documents totaling 4,869 pages, drawn from 8 domains including finance, energy, government, legal, real estate, healthcare, automotive, and supply chain. These include born-digital PDFs, scanned page images, and handwritten forms. There are 67 distinct document types, each associated with a single schema specifying 159 fields on average. Documents are tagged independently along five axes: task challenge (long lists, needle-in-haystack, dense docs), perception challenge (rotated/image-only, scanned, handwriting), table structure (merged headers, cross-page tables, enormous tables, etc.), length (short ≤10 pages, medium 11–50, long >50), and domain. Data provenance differs by type: real documents use ensemble agreement of multiple extractors plus adjudication; synthetic long lists are generated programmatically with known ground truth; scanned forms are human-verified.

3. Architecture / Algorithm: ExtractBench does not provide a single model but evaluates 14 state-of-the-art systems spanning three categories: Commercial VLMs that directly generate structured output from document+schema in a single call; Coding Agents that iteratively use tool APIs to parse, validate, and revise extraction; and Specialized APIs offering dedicated document workflows including grounding. Notable systems include Gemini 3.5 Flash, GPT-5.4 Nano, Codex GPT-5.5, Claude Code Opus 4.8, and multiple LlamaExtract variants. The benchmark requires output to conform to the input JSON schema, including nested records, arrays, nullable fields, and value constraints.

4. Training Regime: Not applicable since this is an evaluation benchmark paper. Models tested are off-the-shelf or recent frontier methods generally trained on large corpora but no fine-tuning specific to ExtractBench is done.

5. Evaluation Protocol: Metrics include unified value F1 measuring exact normalized matches of individual scalar fields or cells in records, computed per document and averaged unweighted. Order-insensitive matching for repeated records uses Hungarian assignment to align predicted and ground-truth records minimizing mismatches. Visual grounding metrics include word-level grounding F1 requiring correct value and a predicted bounding box overlapping the verified ground truth box at IoU ≥0.5, and a weaker page-level grounding F1 requiring correct value and only correct page assignment. Cost per page in cents is calculation based on actual token usage or credits from API providers. The evaluation is done on the full 370 docs and stratified by tags/axes for slice analysis. No adversarial or distribution shift testing beyond challenging document variations is reported. Cross-validation is not applicable.

6. Reproducibility: Ground-truth data, schemas, and evaluation code are publicly released on HuggingFace and GitHub repositories linked in the paper. The benchmark includes detailed annotation and ground-truth generation workflows, supporting adoption and extension. Model weights for proprietary commercial systems are not public. Some open-source systems evaluated are included without fine-tuning code.

Example End-to-End: To evaluate a system on a given document and its JSON schema, the system is given the full document (born-digital PDF or scan) and the schema specifying all 159 fields. The system outputs a schema valid JSON object with values and bounding boxes for each field. The benchmark evaluates exact normalized matches for these fields compared to the ground truth, accounting for repeated records via an optimal assignment. The bounding boxes are compared to verified boxes with IoU ≥0.5 to compute word-level grounding. The system’s per-page cost is computed. These scores contribute to aggregate metrics and per-challenge tag breakdowns.

Technical innovations

• Comprehensive schema-guided extraction benchmark combining value accuracy, record completeness at scale, visual grounding at word and page level, and economic cost evaluation.
• Multi-axis challenge tagging (task, perception, table structure, length, domain) enabling precise failure mode diagnostics across 370 real and synthetic enterprise documents.
• Scalable multi-source ground truth pipeline combining ensemble agreement (for real documents), programmatic generation (for synthetic long lists), and human verification (for scanned forms).
• First published benchmark evaluating both structured extraction and associated visual grounding for enterprise-grade documents spanning scanned and handwritten sources.

Datasets

• ExtractBench — 370 documents, 4869 pages — mix of real born-digital, scanned, handwriting, and synthetic documents; publicly released via HuggingFace and GitHub

Baselines vs proposed

• Commercial VLMs (e.g. Gemini 3.5 Flash): overall unified value F1 = ~79.8% vs LlamaExtract Agentic Plus: 95.6%
• Coding Agents (Codex GPT-5.5): overall unified value F1 = 93.6% at cost 27.8¢/page vs LlamaExtract Agentic Plus: 95.6% at 8.1¢/page
• Reducto Deep Extract (specialized API): overall unified value F1 = 90.4% vs LlamaExtract Agentic Plus: 95.6%
• Grounding word-level F1: LlamaExtract Agentic Plus = 46.4% vs Reducto Deep = 43.3%, others ≤25%; page-level grounding F1: LlamaExtract Agentic Plus = 84.9% vs Reducto Deep = 71.7%
• On document length slices, commercial VLM F1 drops from 87.9% (short) to 27.9% (long), while LlamaExtract Agentic Plus remains over 90% across lengths

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.29677.

Fig 1: ExtractBench scores schema-guided extraction on real enterprise documents: given a document and its

Fig 3: How ground truth is constructed, one strip per source type. A card’s border marks who runs the step (blue:

Fig 2: ExtractBench coverage across the five tag axes. Each bar is the share of the 370 documents carrying the tag,

Fig 4: Overall unified value F1 versus mean document-level cost per page, pooled over every scored document.

Fig 5 (page 5).

Limitations

• No adversarial robustness evaluation against intentional data poisoning or obfuscation attacks.
• Cost estimates rely on provider-published pricing as of July 2026; actual costs may vary with volume or negotiated contracts.
• Distribution shift testing beyond label-verified natural variations in documents is limited; unseen schemas or domains could degrade accuracy.
• Ground truth for many real documents depends on ensemble agreement and adjudication rather than full manual annotation.
• Open-source models tested without task-specific fine-tuning may under-represent achievable performance with adaptation.
• Benchmark emphasizes labeling accuracy and cost but does not evaluate model latency or resource usage comprehensively.

Open questions / follow-ons

• How can extraction agents better handle extremely large documents without truncation or recall loss, possibly through iterative or memory-aware methods?
• Can models improve grounding quality, increasing word-level grounding F1 while maintaining high value accuracy?
• What are effective strategies to adapt schema-guided extraction agents dynamically to novel unseen schemas or business domains?
• How does fine-tuning or in-context learning on ExtractBench’s diverse documents improve frontier models’ accuracy and robustness?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, ExtractBench highlights the challenge of schema-guided extraction from complex real-world documents—often a vector for automated form filling or document scraping bots. The benchmark's multi-dimensional tags and grounding metrics provide a framework to evaluate extraction accuracy together with traceability, instrumental in detecting automated agents versus humans. The cost metrics reinforce the economic constraints in scaling extraction defenses. Systems that handle long, noisy, or handwritten documents robustly while providing source traceability (e.g., bounding box grounding) could inform anti-automation strategies that rely on fine-grained document understanding. Furthermore, the gap observed in commercial VLMs’ performance on long or dense documents versus coding agents suggests exploitable weaknesses or detection points in automated extraction pipelines.

Cite

@article{arxiv2607_29677,
  title={ ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction },
  author={ Boyang Zhang and Adrian Lyjak and Eli Stewart and Zhaoqi Li and Simon Suo },
  journal={arXiv preprint arXiv:2607.29677},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.29677}
}

Read the full paper

• arXiv:2607.29677 (abstract)
• arXiv:2607.29677 (PDF)