Task-Specific Multimodal Question Answering Agents via Confidence Calibration and Incremental Reasoning for QANTA 2026
Source: arXiv:2607.09623 · Published 2026-07-10 · By Nirjhar Das, Md. Al-Mamun Provath
TL;DR
This paper describes the winning submission to the QANTA 2026 shared challenge on efficient multimodal question answering in a quizbowl setting. QANTA requires systems to answer pyramid-style questions with incrementally revealed textual clues and supporting images, operating under strict latency and computational constraints. Two distinct tasks are evaluated: Tossup questions, demanding well-calibrated, timely buzzing decisions under uncertainty, and Bonus questions, emphasizing accurate, structured answers and human-AI collaboration. The authors propose a task-specific two-agent architecture: a GPT-4.1-mini-based Tossup agent focusing on confidence-calibrated incremental answering and numeric reasoning guardrails, and a GPT-4.1-based Bonus agent optimized for leadin-aware reasoning, multimodal evidence fusion, and succinct evidence explanations to aid human adoption. Without relying on retrieval or ensemble methods, the system emphasizes lightweight reasoning policies and confidence calibration within hosted API constraints. Their approach achieved the highest overall leaderboard score of 0.402 (Tossup score 0.238, Bonus Effect 0.164), demonstrating that carefully crafted task-specific reasoning strategies and confidence calibration are critical for resource-constrained, incremental multimodal question answering.
Key findings
- Using a GPT-4.1-mini Tossup agent with calibrated confidence and a Numeric Firewall reduced overconfident buzzing on isolated quantitative clues, improving Tossup expected score to 0.238 with 72.5% buzz precision and 94.2% buzz frequency.
- Bonus agent using full GPT-4.1 with leadin-aware reasoning and multimodal evidence routing achieved 89.1% part accuracy, 72.7% question accuracy, 88.2% calibration, and 33.8% human adoption rate, contributing a Bonus Effect of 0.164.
- Overall combined system scored 0.402 on the leaderboard, surpassing the closest competitor by 0.032, illustrating benefits of task specialization and confidence calibration.
- Confidence threshold selection was crucial; a buzz confidence cutoff of ≥ 0.90 balanced early answering against avoiding costly errors (Table 4), achieving 100% precision on development set and best expected score.
- Distinctive clues with unique entity identifiers produced more stable hypotheses than ambiguous generic early clues, reducing related-entity confusion errors.
- Selective late-fusion multimodal reasoning routed visual evidence mainly for disambiguation rather than primary hypothesis generation, preserving computational efficiency.
- Numeric Firewall domain-specific guardrail prevented increase of confidence from isolated numbers unless supported by contextual anchors, improving quantitative reasoning robustness.
- Simpler prompting strategies outperformed heavily engineered prompting in both Tossup and Bonus agents regarding calibration and adoption.
Threat model
The adversary in this setting corresponds to the quizbowl question design process, which aims to challenge the QA system with incremental clues, potentially ambiguous or related entities, and multimodal distractors (images). The system must avoid overconfident premature answer commitments (buzzes) that incur penalties. The adversary cannot tamper with the hosted API environment but tests system robustness through challenging linguistic and visual ambiguities under real-time constraints.
Methodology — deep read
The authors developed a two-agent system specifically tailored to two tasks in QANTA 2026: Tossup (incremental buzzer decisions under uncertainty) and Bonus (full-context structured answering). The threat model involves a hosted evaluation environment with adversarial distractors inherent in the quizbowl setting, where models must avoid premature or incorrect buzzes.
Data consists of multimodal quizbowl questions with pyramidally revealed textual clues and optional images spanning domains such as literature, history, science, fine arts, and current events. The Tossup task requires incremental processing with streaming clues; Bonus tasks provide complete context before answering. No external retrieval pipelines or ensembles were used; both agents accessed via hosted GPT-4.1 APIs.
Architecture: The Tossup agent uses GPT-4.1-mini, optimized for low-latency incremental reasoning. It outputs an answer prediction plus a verbalized numeric confidence estimate P(correct). Confidence calibration uses a threshold-based expected value (EV) buzzing policy with EV = P(correct) - 0.5 * P(incorrect). Buzzes are issued only if confidence ≥ 0.90 to avoid overconfident errors, especially on quantitative clues.
A Numeric Firewall guardrail was implemented as a prompt-level reasoning constraint preventing confidence above a baseline if evidence is isolated numbers without corroborating context (named entities, formulaic context, or multiple consistent clues).
The Bonus agent uses full GPT-4.1 to process complete question context, including the leadin (theme/category guiding reasoning) and multimodal evidence. It applies leadin-aware structured relational reasoning and late-fusion multimodal evidence routing: initial textual hypotheses are cross-checked against visual evidence to adjust confidence up/down or discard contradictory candidates.
For human-AI collaboration, the Bonus agent outputs concise, confidence-calibrated answers with brief evidence explanations to aid trust and adoption.
Training regime details are not specified; both agents accessed as hosted API endpoints during official evaluation. Prompt engineering was iterative, focusing on simpler, robust prompting over heavy engineering.
Evaluation metrics differ for Tossup (expected score, buzz precision/frequency/position, win rate, latency, cost) and Bonus (part/question accuracy, calibration, adoption, bonus effect). Held-out test data were incrementally revealed in live challenge environment. Ablations include confidence thresholds and guardrail effects.
Reproducibility depends on using hosted GPT-4.1 and GPT-4.1-mini APIs and available competition datasets; code release is not mentioned. The system demonstrates that efficiently calibrated decision policies and domain-specific guardrails provide gains without heavier models or retrieval.
Technical innovations
- A task-specific two-agent architecture that separately optimizes for incremental buzzing (Tossup) and structured complete-context answering (Bonus) in multimodal quizbowl QA.
- A calibrated confidence-based Expected Value buzzing policy with a conservative decision threshold (≥ 0.90) for reliable incremental answer commitment.
- The Numeric Firewall: a prompt-level numeric reasoning guardrail that prevents overconfident predictions on isolated numeric evidence unless corroborated by contextual anchors.
- Leadin-aware reasoning in complete-context Bonus questions, leveraging the thematic leadin to guide structured relational reasoning and consistent answer selection across parts.
- Selective late-fusion multimodal evidence routing, integrating visual information primarily for disambiguation and confidence adjustment rather than equal modality weighting.
Datasets
- QANTA 2026 challenge dataset — size not explicitly stated — hosted live at ICML 2026 Workshop EMM-QA
Baselines vs proposed
- Closest competing system: Overall Score = 0.370 vs proposed: 0.402
- Tossup buzz precision: other participants < 72.5% vs proposed: 72.5%
- Tossup buzz frequency: other participants < 94.2% vs proposed: 94.2%
- Bonus part accuracy: competitor unspecified vs proposed: 89.1%
- Bonus adoption: competitor unspecified vs proposed: 33.8%
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.09623.
Fig 1: Overview of the proposed QANTA 2026 system. The Tossup agent combines evidence routing, confidence calibration, a
Limitations
- The approach depends heavily on prompt engineering and access to large hosted GPT-4.1 APIs; hence, reproducibility outside this environment may be limited.
- No explicit adversarial robustness or stress testing against adaptive attackers was conducted.
- Numeric Firewall relies on heuristic contextual anchors which might fail on novel or ambiguous quantitative clues.
- No detailed analysis of performance across different question domains or image types was provided.
- Human adoption rate in Bonus task is moderate (33.8%), indicating room for improving trust and interpretability.
- The system does not incorporate retrieval or ensemble-based enhancements, potentially limiting recall capacity.
Open questions / follow-ons
- How can entity disambiguation be further improved in incremental question answering to reduce related-entity confusion under incomplete evidence?
- Could adaptive confidence estimation mechanisms that evolve with question progress surpass fixed thresholds like the 0.90 cutoff currently used?
- What retrieval or external knowledge integration methods could complement the proposed lightweight reasoning policies without violating efficiency constraints?
- How can multimodal evidence integration be made more dynamic, weighting textual and visual modalities adaptively per question and clue?
Why it matters for bot defense
From a bot-defense and CAPTCHA perspective, this work demonstrates the benefits of finely tuned confidence calibration and reasoning guardrails in systems that must act under uncertainty and incremental information revelation. Techniques like confidence-based selective prediction and domain-specific error mitigation (e.g., numeric reasoning guardrails) may inspire approaches to distinguish human-like deliberation from superficial pattern matching in CAPTCHA-solving bots. Moreover, the evidence-routing and late-fusion multimodal integration strategies underline the importance of judicious resource use and selective attention, which can inform defensive architectures designed to detect and limit automated exploitations relying on multimodal inputs. Finally, the detailed error analysis highlights the challenges posed by carefully crafted ambiguous inputs, underscoring the value of explicit confidence calibration when evaluation penalties for errors are high.
Cite
@article{arxiv2607_09623,
title={ Task-Specific Multimodal Question Answering Agents via Confidence Calibration and Incremental Reasoning for QANTA 2026 },
author={ Nirjhar Das and Md. Al-Mamun Provath },
journal={arXiv preprint arXiv:2607.09623},
year={ 2026 },
url={https://arxiv.org/abs/2607.09623}
}