Uncertainty-Aware Generation and Decision-Making Under Ambiguity

Source: arXiv:2606.30578 · Published 2026-06-29 · By Nico Daheim, Iryna Gurevych

TL;DR

This paper addresses the challenge of integrating uncertainty awareness into generation and decision-making processes of large language models (LLMs) when applied to inherently ambiguous, subjective, and multi-step tasks such as tutoring and peer reviewing. The key novelty lies in leveraging Bayesian decision theory combined with conformal prediction sets to explicitly capture and marginalize over uncertainty in discrete latent decision states—namely tutoring strategies or reviewer scores—when generating textual outputs. The authors propose methods to compute Bayes-optimal decisions that maximize expected utility over a distribution of plausible discrete states and also explore risk-averse minmax approaches that optimize worst-case utilities within conformal prediction sets. Empirical evaluation on multiple datasets (Review-5K, NLPEER, MathDial) and various LLMs (Gemma3-27B, Qwen3-30B, Mistral-3.2-24B, TutorRL-7B) demonstrates that Bayesian decision rules consistently improve generation utility metrics compared to naive sampling, with conformal prediction-based pruning yielding computational savings and added statistical guarantees. However, risk-averse minmax strategies can degrade performance by favoring overly generic outputs when prediction sets are large due to high ambiguity. This work represents an important systems-level advance on decision-making with LLMs under ambiguity, highlights limitations in current categorical state prediction quality, and outlines open challenges for uncertainty-aware modeling and dataset construction for subjective tasks.

Key findings

• Bayesian decision-making rules (maximizing expected utility over score or strategy distributions) improve generation utility metrics such as actionability, helpfulness, groundedness, and verifiability on Review-5K and NLPEER datasets compared to sampling baselines (e.g., Table 1 shows average increases in actionability from 1.99 to 2.14 for Gemma3 on ICLR reviews).
• Conformal prediction reduces the number of utility calculations by pruning the state space (e.g., compressing prediction sets to 47-76% of total label space, Table 3), enabling computational savings with minimal performance hit.
• Risk-averse minmax decision rules can underperform Bayesian approaches, especially with wide prediction sets, often producing generic or low-utility outputs (Table 2 shows minmax reward is 0.809 vs 0.892 for Bayes on MathDial tutoring).
• LLM-based classifiers predicting discrete tutoring strategies or review scores have modest accuracy (~30-47%), evidencing high label ambiguity and prediction difficulty.
• Increasing candidate sample sets |H| for Bayes decisions improves metrics consistently (Figure 2), while minmax rules degrade with larger |H| due to interaction effects with prediction set size.
• Using multiple labels per example for conformal calibration (Monte-Carlo conformal prediction) better accounts for label ambiguity and is recommended but existing datasets lack sufficient multi-label annotations.
• Different LLMs exhibit varied performance tradeoffs: Gemma3 and Mistral3 perform similarly, Qwen3 shows weaker actionability but better other metrics, likely due to post-training.
• Utilizing separate reward models (independent from utility models) improves evaluation reliability over prompted LLM judgment.

Threat model

Adversaries are not explicitly modeled as attackers but implied ambiguities arise from inherent subjectivity, label noise, and disagreement among human annotators (reviewers, tutoring experts). The threat is poor LLM decisions caused by uncertain latent states leading to low-utility or misleading outputs. The methods aim to model and mitigate this uncertainty rather than adversarial manipulation. The adversary cannot alter the underlying distributions or data generation but practical robustness depends on classifier and utility estimator quality.

Methodology — deep read

The authors start from a threat model of an ambiguous decision environment where the true latent decision state (e.g. tutoring strategy or review score) is unknown and multi-modal, and where human labels disagree significantly. Adversaries or noise causing input ambiguity are implicit. The goal is to generate reliable textual outputs that account for this uncertainty.

Data comes from three main datasets: Review-5K (ICLR paper reviews), NLPEER (ACL Rolling Review peer review data), and MathDial (dialog tutoring with student attempted solutions). Splits reserve 90% training and 10% calibration. Labels include human review scores and tutoring strategies; multiple human annotations exist only for review datasets.

Algorithms are centered on Bayesian decision theory: given an input x (e.g. paper text or dialog context), and a discrete latent state space Ω (e.g. possible scores or tutoring strategies), the method models a distribution p(ω|x) over Ω predicted by a finetuned categorical classifier (LLama-3.2-3B-Instruct or similar). The decision rule chooses output y* in output space V* that maximizes the expected utility u(y,ω) weighted by p(ω|x).

Utility functions u encode application-specific quality/relevance criteria, evaluated by separate LLM reward models not the same as generator or classifier. Utility estimation is approximated by Monte Carlo sampling M candidate outputs (H) from p(y| x), scored against ω.

Because Ω can be large, conformal prediction is used to define a calibrated set C(x)⊆Ω of plausible states containing the true label with coverage 1-α (α=0.05). This prunes the summation over Ω in expected utility to only ω in C(x). The conformal calibration uses split data and nonconformity scores 1-p(ω | x). For multi-label cases, Monte-Carlo conformal prediction is employed to handle ambiguity and label variation.

They also implement a risk-averse minmax alternative, choosing output y to maximize worst-case utility over ω in C(x) (Eq. 7), valuing cautious decisions.

Model training involves finetuning Llama-3.2-3B-Instruct on score/stategy classification tasks with 90% training data. Sampling sizes are |H|=64 for reviewing and 32 for tutoring (varied up to 192 in some experiments). Generation models include Gemma3-27B, Qwen3-30B, Mistral-3.2-24B, and TutorRL-7B.

Evaluation metrics include various human-aligned utility measures (Actionable, Verifiable, Helpful, Grounded) scored by external reward models independent of generation or classification LLMs to avoid bias. They report averages over multiple random seeds and use ablations comparing sampling, Bayes, Bayes+conformal pruning, and minmax decision rules.

Experiments show Bayes expected-utility maximization clearly outperforms naive sampling baselines and is more robust to prediction set size and ambiguity. Conformal prediction pruning significantly reduces computation by shrinking Ω with minimal performance loss. Minmax decisions perform well only when prediction sets are narrow, otherwise can optimize for generic low-utility outputs.

The authors provide code and detailed prompts. Some datasets like MathDial lack multiple annotations needed for strong conformal guarantees in ambiguous settings, which the authors note as a reproducibility and methodological limitation. The approach requires good classifiers for p(ω|x) and strong reward models for u(y,ω) evaluation.

Technical innovations

• Application of Bayesian decision theory to LLM generation tasks by marginalizing over distributions on discrete latent decision states (tutoring strategies, review scores) to maximize expected utility.
• Integration of conformal prediction to generate calibrated prediction sets over categorical decision states to provide statistical coverage guarantees and prune computations in expected utility maximization.
• Use of risk-averse minmax decision rules over conformal prediction sets to optimize worst-case utility, exploring its tradeoffs with Bayesian averaging under high ambiguity.
• Adaptation of Monte-Carlo conformal prediction for settings with label ambiguity and multiple plausible labels to better handle human label variation in subjective tasks.
• Empirical demonstration of how these uncertainty-aware decision rules improve concrete utility metrics across multiple real-world ambiguous generation tasks.

Datasets

• Review-5K — ~5,000 paper reviews from ICLR [public]
• NLPEER — Peer review dataset from ACL Rolling Review [public]
• MathDial — Dialog-based tutoring data with math problem solving [public but limited multi-labelings]

Baselines vs proposed

• Sampling baseline: actionability = 1.99±0.008 (Gemma3 on ICLR reviews) vs Bayes decision: 2.14±0.005
• Sampling baseline: scalar reward = 0.858±0.002 (TutorRL-7B MathDial) vs Bayes decision: 0.892±0.002
• Minmax decision (risk-averse) reward = 0.809±0.003 vs Bayes reward 0.892±0.002 on MathDial tutoring
• Classification accuracy for state prediction: 30.1%–32.6% on review datasets, 47.3% on MathDial tutoring classifier
• Oracle classifier on Review-5k achieves 58.6% accuracy but no major downstream gain
• Conformal prediction compresses label space by 24-53% reducing utility computation by up to 50% with minimal quality drop

Limitations

• Classifier accuracy for state predictions (scores, strategies) remains low (~30-47%), limiting decision quality and uncertainty calibration.
• Existing datasets lack sufficient multiple annotations per input to properly calibrate conformal prediction sets under label ambiguity.
• Risk-averse minmax decision strategy underperforms with wide prediction sets by producing overly generic or low-utility outputs.
• Utility functions and reward models employed are task-specific and require development and validation, limiting direct generalization.
• The approach relies on sampling candidate outputs (|H| up to 192), which is computationally intensive.
• LLM probabilities and calibration assumptions (exchangeable data, single ground-truth label) may be violated in real ambiguous settings, affecting guarantees.
• No direct evaluation under adversarial or distribution shift scenarios that commonly affect security or bot-defense tasks.

Open questions / follow-ons

• How to improve classifiers for latent state prediction (scores, tutoring strategies) especially under label ambiguity and multi-label conditions?
• How can datasets be constructed with multiple high-quality annotations per example to enable rigorous conformal calibration under human disagreement?
• How to design utility functions and reward models that better represent user goals and pedagogical or reviewing principles across diverse contexts?
• What are effective ways to integrate uncertainty-aware decision-making methods into interactive systems without overwhelming users with prediction set information or ambiguous outputs?

Why it matters for bot defense

While not directly focused on bot or CAPTCHA defense, this work provides valuable insights for bot-defense practitioners interested in uncertainty-aware decision-making under ambiguous inputs. The Bayesian and conformal prediction methods offer ways to quantify and leverage uncertainty over discrete latent states or user intents, which is relevant for behavioral or challenge-response modeling in CAPTCHAs. The approach to statistically guarantee coverage over uncertain decision sets could help build more trustworthy and interpretable bot detection or challenge generation systems that must operate under noisy or adversarial signals. However, the computational cost and requirement for strong classifiers may pose practical challenges. Practitioners might consider adapting these concepts to tune challenge difficulty or interpret ambiguous host signals more rigorously. The paper also highlights the difficulties of managing multiple plausible labels and the risks of overly conservative (risk-averse) decision rules, which could inform challenge design to avoid generic or unhelpful outputs.

Cite

@article{arxiv2606_30578,
  title={ Uncertainty-Aware Generation and Decision-Making Under Ambiguity },
  author={ Nico Daheim and Iryna Gurevych },
  journal={arXiv preprint arXiv:2606.30578},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.30578}
}

Read the full paper

• arXiv:2606.30578 (abstract)
• arXiv:2606.30578 (PDF)