Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness
Source: arXiv:2607.19322 · Published 2026-07-21 · By Xilun Chen, Zhaleh Feizollahi, Ross Goodwin, Seungwhan Moon, Scott Yih, Pinar Donmez et al.
TL;DR
This paper addresses the underexplored problem of evaluating factual completeness in long-form open-ended generation, a complementary dimension of factuality to the commonly studied precision. Existing evaluation approaches predominantly decompose answers into independent atomic fact checks to verify correctness, which misses whether a response contains all the relevant information expected for a complete answer. The authors propose a novel two-level meta-rubric framework where a structured, human-verified meta-rubric captures the organization, importance, ordering, and grouping of facts needed for completeness. This meta-rubric is mechanically compiled into a flat binary checklist for reliable machine grading by an LLM judge. They instantiate this as GAMUT, a new large-scale benchmark of 1,813 multimodal questions across 10 everyday domains grounded in real wearable imagery, each paired with expert-verified evidence-based rubrics. Extensive evaluation of 14 state-of-the-art proprietary and open-weight models shows GAMUT is challenging and highly discriminative, with best models only reaching 58.7% on the GAMUT score, revealing large factual recall gaps not exposed by precision-focused metrics. The framework is modality-agnostic, and a text-only variant is also released. Results demonstrate that the new completeness-focused evaluation and structured rubric design better capture knowledge recall quality in open-ended generation.
Key findings
- GAMUT benchmark contains 1,813 questions grounded on real wearable images spanning 10 diverse domains.
- Best model (Gemini 3.1 Pro) scores only 58.7% on the GAMUT score, indicating room for improvement in factual completeness.
- GAMUT score decomposes into tiers: Answer-Critical, Valuable, and Context with descending importance weights.
- Judgments by different LLM judges (Gemini 3.1 Pro, Claude Opus 4.8, Qwen3-VL 235B) produce consistent model rankings with under 1.8% score variation.
- Dominant modeling failure mode is omission rather than contradiction, with strongest models missing over 25% of rubric elements and weaker models missing over 66%.
- Converting meta-rubrics to binary checklists enables low-variance, scalable automated scoring by LLM judges while retaining expressive structured evaluation.
- Evaluating models on text-only variant (1,806 questions) raises scores by about 10–20 points consistently, isolating the complexity of visual identification from knowledge completeness.
- The two-level meta-rubric design captures coverage, ordering, relationships, and importance of facts beyond flat boolean fact-checking.
Threat model
The adversary is a generation model producing open-ended long-form answers to grounded questions; it may omit facts or introduce contradictions. The evaluation system assumes the judge (an LLM) can reliably score independent rubric items but does not consider adversarially crafted responses designed to fool the scorer or perturb rubric structure.
Methodology — deep read
The paper's approach aims to measure factual completeness—a notion that assesses whether an answer contains all necessary facts in coverage, order, and relational structure—contrasting with prior fact-checking that treats facts as independent boolean items.
Threat model or assumptions focus on assessing model-generated long-form answers to open-ended, real-world grounded questions from wearable imagery; the adversary is effectively an automated grading system (an LLM judge) striving for consistent and reliable evaluation without holistic variance.
Data originates from the CRAG-MM dataset, comprising 1,938 egocentric wearable-device images from 10 everyday domains (plants, food, vehicles, objects, etc.). After iterative LLM and multi-round human expert annotation revisions with strict quality controls, 1,813 questions were finalized, each requiring multi-paragraph deep-research style answers supported by web evidence.
The evaluation rubric has two levels: a rich structured meta-rubric that organizes facts into typed components (Simple Knowledge, Strict List, Flexible List, Process, Relationship) with importance tiers (Answer-Critical, Valuable, Context) capturing coverage and ordering; and a deterministic conversion algorithm mechanically compiling this into a flat checklist of pass/fail binary rubric items that an LLM judge can score independently for reliability.
Meta-rubric construction involved LLM-generated candidate rubrics grounded in human-verified web evidence snippets, followed by LLM self-refinement (search and audit phases), and multi-round expert human revision directly on the binary checklists to ensure completeness and correctness.
During evaluation, 14 large language and multimodal models produce free-form answers given image-question pairs (or text-only question variants). For each response, an LLM judge evaluates the binary checklist items independently, assigning one of four verdicts (meets, partially meets, missing, contradicts). These are converted to numeric scores with penalties for contradiction and combined by importance-weighted averaging across tiers to produce an overall GAMUT score.
Evaluation also explored judge agreement by rescoring all models with three different LLM judges, showing robust rankings and scoring consistency. A text-only variant rewrote questions explicitly naming the subject to isolate visual identification effects.
All code and data are publicly released to facilitate reproducibility. The paper provides detailed examples of a full meta-rubric, the compiled checklist, and scoring process on representative questions.
Technical innovations
- Introduction of a two-level meta-rubric framework that captures rich structure (importance, ordering, relationships, coverage thresholds) in open-ended factual completeness evaluation.
- Mechanical conversion of expressive structured meta-rubrics into flat binary checklists for consistent, low-variance automated scoring by LLM judges.
- Creation of the first large-scale, multimodal everyday deep-research benchmark (GAMUT) grounded in real wearable imagery, combining LLM generation with multi-round expert verification.
- A scoring scheme distinguishing partial meets and contradictions and weighting rubric items by annotated importance tiers to better reflect answer quality.
- A modality-agnostic evaluation framework enabling both multimodal and text-only variants of the benchmark.
Datasets
- GAMUT — 1,813 open-ended wearable-imagery grounded questions with human-verified structured rubrics — constructed from CRAG-MM imagery and verified web evidence.
- CRAG-MM — approx. 1,938 wearable-device images across 10 everyday domains, source for grounding GAMUT.
Baselines vs proposed
- Gemini 3.1 Pro: GAMUT score = 58.7% vs others ranging down to Llama 3.2 11B at 5.1%.
- In text-only setting, Gemini 3.1 Pro score raises to 74.1%, a +15.4 point gain over multimodal.
- Judges Gemini 3.1 Pro and Claude Opus 4.8 rank models identically with scores within 1.8%.
- Lower performance across open-weight models (e.g., Qwen3-VL 235B at 33.0%) demonstrates task difficulty.
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.19322.
Fig 1: A representative example (jollof rice) from Gamut (some rubrics not shown for brevity). Top: an everyday
Fig 5: A complete example from Gamut (a Tang sancai figure), exercising all five meta-rubric types including a
Limitations
- The benchmark focuses on coverage and ordering but may not capture all nuanced linguistic quality or subjective answer style.
- Evaluation relies solely on LLM judges designed for rubric-based grading; human judge consistency or disagreement on borderline cases is not deeply analyzed.
- Visual information contributes a roughly constant difficulty gap but the interplay between perception and knowledge completeness could be further explored.
- Rubrics, while verified extensively, are still constructed using LLMs aided by experts, possibly introducing bias or incompleteness.
- The scored questions are grounded in wearable imagery contexts, limiting immediate generalization to other multimodal or purely textual knowledge domains.
- No explicit adversarial or stress testing of the rubric or judge robustness against adversarially generated responses.
Open questions / follow-ons
- Can this two-level meta-rubric framework be adapted for other domains beyond wearable imagery, such as medical or scientific knowledge?
- How well do human raters agree with the LLM judge scores and rankings on GAMUT, especially on partial meets and contradictions?
- Can the rubric be extended to dynamically adapt to novel acceptable answer variants or updated factual knowledge?
- What are optimal methods for designing and weighting meta-insights and relational rubric items to balance expressivity and grading reliability?
Why it matters for bot defense
For bot-defense and CAPTCHA practitioners, GAMUT presents a novel evaluation paradigm demonstrating that factual completeness in open-ended generation requires richer structured evaluation metrics beyond simple precision-based fact-checks. In bot-detection contexts where long-form user-generated content or question answering is relevant, adopting a similar two-level rubric approach could enable more reliable automatic detection of incomplete or partially truthful generative outputs. The mechanical conversion from expressive rubrics to binary checklist items scored by LLM judges offers a practical way to maintain grader reliability without sacrificing nuanced analysis of answer structure and coverage. However, practitioners should note that scoring remains challenging (best models <60%), omission errors dominate, and differences in visual-perception components can confound pure knowledge assessment. Applying GAMUT insights calls for careful rubric design tailored to domain-specific knowledge structures and potential multimodal inputs. The released codebase also offers a promising starting point for building completeness-aware evaluation workflows applicable to various content moderation or anti-bot verification tasks involving generation.
Cite
@article{arxiv2607_19322,
title={ Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness },
author={ Xilun Chen and Zhaleh Feizollahi and Ross Goodwin and Seungwhan Moon and Scott Yih and Pinar Donmez and Babak Damavandi and Luna Dong },
journal={arXiv preprint arXiv:2607.19322},
year={ 2026 },
url={https://arxiv.org/abs/2607.19322}
}