CoMet: Context and Multiplicity Decomposition for Multimodal Uncertainty Estimation

Source: arXiv:2606.32012 · Published 2026-06-30 · By Sanghyuk Chun, William Yang, Amaya Dharmasiri, Olga Russakovsky

TL;DR

This paper addresses the challenging problem of uncertainty estimation in multimodal large language models (MLLMs), especially for open-ended tasks where outputs are unbounded and ambiguous. The authors propose CoMet, a novel uncertainty estimation framework that decomposes predictive uncertainty into two complementary components: context-specific uncertainty which reflects ambiguity induced by the given task or prompt context, and multiplicity-specific uncertainty which captures how many plausible answers compatible with the input remain within that context. This decomposition is motivated by a newly introduced binary matching variable indicating semantic compatibility between an input, a candidate answer, and the context. To estimate these uncertainties efficiently, CoMet uses a lightweight post-hoc module trained with approximate matching probabilities obtained by prompting MLLMs as verifiers and approximating the context-conditioned answer distribution from a clustered candidate answer set. Importantly, CoMet requires no autoregressive answer generation or expensive repeated sampling at inference, enabling more practical uncertainty estimation for open-ended multimodal settings.

Extensive experiments across multiple large-scale open-ended visual question answering benchmarks (VQA v2, VizWiz, OK-VQA), hallucination detection tasks, and multiple-choice VQA datasets (MMMU, MMMU Pro, MMStar) with diverse MLLM backbones demonstrate that CoMet consistently outperforms state-of-the-art generation-based, sampling-based, perturbation-based, and candidate-answer baselines. Ablation studies confirm the complementary value of the two uncertainty components. Qualitative analysis further illustrates how context and multiplicity uncertainty provide interpretable insights into prediction ambiguity. Overall, CoMet advances uncertainty estimation in MLLMs by combining theoretical rigor, practical deployment efficiency, and strong empirical gains.

Key findings

• CoMet improves AUROC by +0.100 over the strongest baseline P(Correct) on VQA v2 with Qwen3VL-2B backbone.
• CoMet achieves +0.135 AUROC gain over VL Uncertainty [11], a sampling-based method requiring repeated generation.
• On hallucination detection and unanswerability detection, CoMet attains best or second-best performance across backbones.
• For MCQ VQA benchmarks, despite being designed for open-ended tasks, CoMet remains competitive and often outperforms candidate-based baselines.
• Combining both uncertainty terms (context-specific ut and multiplicity ux|t) leads to an average AUROC of 0.798, outperforming models trained with either term alone (0.780 or 0.686).
• Calibrated matching probability p(m=1|x,t,y) estimated via prompting MLLMs as verifiers is effective for uncertainty decomposition.
• Context-conditioned answer distribution p(y|t) constructed via greedy clustering from 200k image-question pairs (with 32 candidate answers each) provides a practical model-agnostic prior.
• Inference is efficient and generation-free; no autoregressive sampling or repeated answer generation required.

Threat model

n/a — as the paper focuses on uncertainty estimation in multimodal AI models rather than security threats or adversarial attacks, a formal threat model is not defined. The paper assumes standard ML usage scenarios where uncertainty quantification aids model reliability assessment.

Methodology — deep read

1. Threat model & assumptions: The adversary and security aspects are not emphasized because this is primarily a model uncertainty estimation work, not a security defense paper. The focus is on estimating uncertainty reflecting ambiguous or incorrect model outputs in multimodal large language models (MLLMs). The formulation assumes an image input X, text context T (e.g., a question), and an answer Y. A binary matching variable M indicates semantic compatibility (m=1 if X and Y match under T). This matching is modeled probabilistically.

2. Data: The authors collect 200,000 image-question pairs from a visual instruction tuning dataset (Cambrian). Each pair is associated with ~32 plausible candidate answers obtained by clustering question prompts and pooling answers within clusters. The dataset is split into 199k training and 1k validation samples for training the uncertainty estimator. Additional public VQA datasets (VQA v2, VizWiz, OK-VQA) and MCQ VQA datasets (MMMU, MMMU Pro, MMStar) are used for evaluation. HallusionBench is used for hallucination detection.

3. Architecture/Algorithm: The key conceptual innovation is decomposing uncertainty u(x,t) into context-specific ut(t) and multiplicity-specific ux|t(x|t) terms, derived by introducing matching variable m and computing Rényi-2 entropy over the match-conditioned posterior p(y|x,t,m=1).

• The matching probability p(m=1|x,t,y) is estimated by prompting the MLLM with a verifier prompt asking if candidate answer y is correct for (x,t), producing a calibrated binary probability via logistic transformation with learned β and γ parameters.

• The context-conditioned answer distribution p(y|t) is approximated nonparametrically by clustering similar questions and collecting plausible answers from a large dataset, avoiding reliance on generative model probabilities to reduce model bias.

• Using these, uncertainty components Z1 and Z2 (expectations of matching probabilities and their squares over p(y|t)) are approximated via averages over candidate answer sets.

• The final uncertainty module is a lightweight post-hoc regressor trained on these approximated quantities to predict uncertainty solely from input x and context t.

4. Training: The uncertainty module is trained with a regression loss on the synthesized targets (Z1, Z2, ut) derived from the candidate answer sets and matching probabilities, using 199k training samples. Hyperparameters (β, γ for calibration; λ for balancing terms) are tuned on validation splits. A small adaptation step is done on benchmark-specific samples (900 from MMMU val and 10k from VQA v2 val).

5. Evaluation: Metrics include AUROC, AUPR, and FPR95 treating uncertainty as a binary classifier distinguishing correct vs incorrect predictions. Comparisons include generation-based (NLL, P(Correct), verbalized confidence), sampling-based (consistency entropy/max probability), perturbation-based, and candidate-based baselines. Evaluations are done on held-out test splits and multiple MLLM backbones (Qwen3VL 2B,4B,8B and InternVL3.5-1B). Multiple ablations (e.g., using only one uncertainty component) analyze decomposition impact.

6. Reproducibility: Code is publicly released at the project GitHub. The matching probability estimators use frozen large MLLMs prompted with verifier queries. The context-conditioned answer sets are constructed from an existing public dataset. Details of architectures, training procedures, and hyperparameters are provided, supporting practical reproduction.

Concrete example: For a given image x and question t, CoMet retrieves a candidate answer set for the question cluster, prompts MLLM verifier on each candidate y to get p(m=1|x,t,y), computes average matching probabilities (Z1, Z2), approximates context uncertainly ut from LM probabilities of candidate answers, and outputs the final uncertainty score without needing to autoregressively generate answers or sample repeatedly, efficiently identifying ambiguous or potentially incorrect model outputs.

Technical innovations

• Introduction of a binary matching variable m to model semantic compatibility between input, answer, and context, enabling uncertainty decomposition.
• Novel decomposition of multimodal uncertainty into context-specific and multiplicity-specific terms via Rényi-2 entropy over match-conditioned posteriors.
• Use of MLLMs as verifiers with calibrated prompts to estimate matching probabilities, bypassing separate module training.
• Construction of context-conditioned answer distributions via clustering-based candidate answer sets, avoiding reliance on generative model output distributions.
• Training a lightweight post-hoc uncertainty module enabling efficient generation-free uncertainty estimation in open-ended multimodal settings.

Datasets

• Cambrian visual instruction tuning dataset — 200,000 image-question pairs with 32 plausible answers each — public
• VQA v2 — 214,000 samples originally, 10,000 subsampled for evaluation — public
• VizWiz — size unspecified in paper but public benchmark
• OK-VQA — public benchmark
• HallusionBench — hallucination detection benchmark
• MMMU, MMMU Pro, MMStar — multiple-choice visual question answering benchmarks

Baselines vs proposed

• P(Correct): AUROC = 0.685 vs CoMet: 0.785 (Qwen3VL-2B on open-ended VQA average)
• VL Uncertainty [11]: AUROC = 0.650 vs CoMet: 0.785 (Qwen3VL-2B on open-ended VQA average)
• Consistency Entropy [10]: AUROC = 0.624 vs CoMet: 0.785 (Qwen3VL-2B)
• Candidate Entropy (MCQ): AUROC = 0.719 vs CoMet: 0.842 (on MMMU average)
• Verb. Confidence [3]: AUROC = 0.591 vs CoMet: 0.785 (Qwen3VL-2B open-ended VQA)
• NLL: AUROC = 0.569 vs CoMet: 0.785 (Qwen3VL-2B open-ended VQA)
• Ablation ut only: 0.686 AUROC vs ux|t only: 0.780 AUROC vs both combined: 0.798 AUROC (Qwen3VL-8B on average)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.32012.

Fig 1: Overview of the proposed multimodal uncertainty. Even for the same image x, uncertainty could

Fig 3: Overview of the proposed COMET framework. (a) We construct a context-conditioned answer

Fig 4: Visualization of uncertainties of various samples.

Fig 4 (page 9).

Fig 5 (page 9).

Fig 6 (page 9).

Fig 7 (page 9).

Fig 8 (page 9).

Limitations

• The method relies on the quality and coverage of candidate answer sets to approximate p(y|t), which may be incomplete or biased.
• Matching probability estimation depends on the verifier MLLM and prompt engineering; generalization to radically different tasks may be limited.
• Rényi-2 entropy approximation involves assumptions (e.g., uniform p(y|t) over candidate sets) that may not hold exactly, introducing approximation error.
• Evaluations primarily focus on VQA and related benchmarks; applicability to other multimodal tasks or domains remains untested.
• No adversarial robustness or worst-case evaluation of uncertainty estimates is presented.
• Recalibration parameters (β, γ) require validation data per backbone, adding some tuning overhead.

Open questions / follow-ons

• How well does CoMet uncertainty estimation generalize to other multimodal tasks beyond VQA, such as image captioning or video understanding?
• Can the approach be extended to incorporate temporal or sequential contexts for uncertainty in multimodal dialogue agents?
• How robust is the verifier-based matching probability estimation under distribution shifts or noisy inputs?
• What are the best techniques to construct or expand the context-conditioned answer distribution p(y|t) for more diverse or rare queries?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, CoMet presents a principled framework for estimating uncertainty in AI models that interact with multimodal inputs and provide open-ended text outputs. This is relevant because reliable uncertainty scores can help detect bot-generated answers that are ambiguous, inconsistent, or hallucinated. CoMet’s decomposition aids in interpreting whether uncertainty arises from ambiguous input context or from multiple plausible compatible answers, which can be useful for adaptive challenge generation or user interaction design.

Moreover, CoMet’s generation-free inference and reliance on lightweight post-hoc modules reduce computational overhead compared to sampling- or generation-heavy uncertainty methods. This efficiency is critical for real-time CAPTCHA or bot defense systems deployed at scale. Bot-defense engineers could apply CoMet-inspired uncertainty estimation to multimodal verification tasks, improving detection of automated or low-confidence behaviors without the latency penalty of repeated generative sampling. However, practitioners should note that the approach depends on constructing representative candidate answer sets which may be domain-specific, and that verifier prompt quality influences reliability.

Cite

@article{arxiv2606_32012,
  title={ CoMet: Context and Multiplicity Decomposition for Multimodal Uncertainty Estimation },
  author={ Sanghyuk Chun and William Yang and Amaya Dharmasiri and Olga Russakovsky },
  journal={arXiv preprint arXiv:2606.32012},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.32012}
}

Read the full paper

• arXiv:2606.32012 (abstract)
• arXiv:2606.32012 (PDF)