When are likely answers right? On Sequence Probability and Correctness in LLMs
Source: arXiv:2606.27359 · Published 2026-06-25 · By Johannes Zenn, Jonas Geiping
TL;DR
This paper investigates the fundamental relationship between sequence probability assigned by large language models (LLMs) and the correctness of their generated outputs. The authors rigorously analyze whether higher sequence probability (the conditional probability of a completion given a prompt) aligns with producing correct answers. They frame decoding methods as ways to shift probability mass towards higher-probability sequences — either locally (token-level) or globally (sequence-level). Through extensive empirical studies across 8 decoding methods, 14 LLM variants spanning Qwen and Olmo3 model families, and 6 benchmark datasets (including MATH500, GPQA, HumanEval, MedQA, MMLU, and IFEval), they quantify this relationship at four levels: across decoding methods, within-method hyperparameter changes, across prompt-answer pairs in a dataset, and across repeated samples for the same prompt.
Their main finding confirms that within a fixed dataset and model, higher sequence probability correlates positively with correctness (especially strongly on tasks like MATH500). However, this relationship rarely translates when manipulating decoding choices or hyperparameters — increasing sequence probability through decoding methods generally does not increase accuracy. At the granularity of multiple responses to the same prompt, sequence probability is an unreliable indicator. Thus, while LLM probability distributions do encode some correctness signal, most decoding improvements cannot be explained by simply seeking sequences with higher model likelihood. This nuanced understanding clarifies when and why decoding methods help accuracy, reveals limits of verifier-free self-improvement relying on probability, and informs practical strategies for decoding and self-consistency aggregation.
Key findings
- Within a fixed dataset and model variant, sequence log-probability correlates positively with correctness across samples, with correlation coefficients (Pearson r) as high as +0.96 on MATH500 and consistently positive on GPQA, HumanEval, MedQA, and MMLU, but negative on IFEval.
- Tuning hyperparameters within a single decoding method increases average sequence probability but does not reliably improve correctness; correlations within-method are often weak or inconsistent (Finding 2).
- Across different decoding methods, sequences with higher log-probability do not consistently correspond to higher accuracy; e.g., no method reliably outperforms low-temperature sampling (LTS) baseline in accuracy despite often yielding higher probabilities (Finding 3).
- Within-sample correlations (examining multiple responses for the same prompt) between probability and correctness are symmetrically distributed around zero with zero mean, indicating sequence probability is a poor correctness signal at this level (Finding 5).
- Correlation between probability and correctness approximately increases with overall base accuracy of the model-dataset pair, suggesting verifier-free self-improvement methods relying on probability require a strong initial model performance (Finding 4).
- Posttrained models show more consistently positive correlation between probability and correctness than base models, which can exhibit diverse and sometimes negative correlations on certain datasets like IFEval.
- Beam search, which seeks highest probability sequences globally, often produces less correct and less probable sequences, possibly due to length normalization issues.
- Power sampling methods consistently generate higher probability sequences compared to baselines but are not reliably more accurate, demonstrating overfitting to model probability alone does not optimize correctness.
Threat model
n/a — The paper does not focus on security adversaries but rather analyzes intrinsic statistical properties of LLM sequence probabilities and correctness under different decoding regimes.
Methodology — deep read
The authors conduct an extensive empirical analysis of the relationship between sequence probability assigned by LLMs and the correctness of generated outputs, structured across four correlation levels: across decoding methods, within a method by hyperparameter variation, within a fixed dataset across prompt-answer pairs, and within repeated samples for the same prompt.
Threat Model & Assumptions: The adversary is not explicitly modeled, but the study assumes an LLM trained on predominantly correct textual data, evaluating how its internal sequence probabilities reflect output correctness under different decoding methods. The adversary here can be viewed as the decoding method or configuration trying to maximize correctness.
Data: Six benchmark datasets are chosen, spanning mathematical reasoning (MATH500), code generation (HumanEval), commonsense/out-of-distribution reasoning (MedQA), instruction following (IFEval), factual knowledge (MMLU), and reading comprehension (GPQA). Data splits and labels are as per original datasets; correctness is a binary signal indicating whether the generated output matches the accepted correct answer.
Models: Experiments cover 14 models from three families—Qwen2.5, Qwen3, and Olmo3—at varying scales from hundreds of millions to 8B parameters, including base and posttrained variants.
Decoding Methods & Hyperparameters: Eight decoding methods studied include local token-level methods like low-temperature sampling (LTS), top-k/p, and ε-sampling, and global sequence-level methods such as beam search, best-of-N (BoN), and power sampling variants (SPS, power-SMC). Each method is tested at multiple hyperparameter settings (e.g., varying temperature, k, p thresholds, beam counts).
Methodological Core:
- Sequence probability is defined as the product of conditional token probabilities given the prompt.
- Local methods modify the next-token distribution at each step (tempering or truncating probability mass).
- Global methods approximate sampling or search over entire sequences to maximize sequence probability, like beam search or power distribution sampling.
- The relationship between probability and correctness is quantified using correlation statistics (Pearson and Spearman) at multiple aggregation levels.
Training & Evaluation: Models are evaluated at inference; no fine-tuning is performed during study. Computation leverages available pre-trained checkpoints. Trajectory lengths and sample sizes are limited due to computational constraints (e.g., SPS uses block sizes of 96 to 384 tokens, BoN uses N=8 to 32 samples). Accuracy metrics per dataset are binary correctness; correlations are computed between log-probabilities and these correctness labels.
Evaluation Protocol:
- Across-method correlation fixes model and dataset, comparing probability and correctness variation by method.
- Within-method correlation sweeps method hyperparameters to analyze how probability and correctness co-vary.
- Within-dataset correlation aggregates correctness and probability statistics across prompt-answer pairs for a fixed method and hyperparameter.
- Within-sample correlation analyzes repeated samples on the same prompt to measure consistency of probability as a correctness signal.
Reproducibility: The paper provides extensive appendices detailing methods and hyperparameters. Model families and datasets are publicly referenced; however, some proprietary or closed-source evaluation specifics (like Qwen and Olmo3 internal models or training data) might limit full replication. Code release is not explicitly stated in the snippet.
Example End-to-End: For Qwen3-8B on the MATH500 dataset using scalable power sampling with block size 192, the authors plot scatter correlation of log-probability vs correctness across samples, finding a near-perfect correlation (Pearson r = +0.96). However, tuning hyperparameters to increase sequence probability in this setting does not consistently improve accuracy on unseen prompts, revealing a disconnect between probability maximization and correctness gains in decoding choices.
Technical innovations
- The paper formalizes decoding methods as ways to maximize sequence probability, distinguishing local token-level methods from global sequence-level methods using a variational objective framework over distributions of sequences.
- Introduces and empirically analyzes power sampling methods (SPS, power-SMC) for sequence-level probability sharpening, providing a principled variational interpretation balancing expected log-probability and entropy.
- Comprehensive multi-level correlation analysis framework (across-method, within-method, within-dataset, within-sample) for rigorously quantifying the probability-correctness relationship in LLM outputs.
- Demonstrates that while higher sequence probability correlates with correctness across prompt-answer pairs within datasets, increasing probability through decoding optimization rarely yields accuracy improvements, clarifying fundamental limits of verifier-free self-improvement.
Datasets
- MATH500 — ~500 problems — public benchmark for mathematical reasoning
- GPQA — size unspecified — public/broadly accessible benchmark
- HumanEval — code generation — ~164 problems — open-sourced
- MedQA — size unspecified — benchmark on commonsense/out-of-distribution reasoning
- MMLU — ~57 tasks with multiple-choice questions — public multi-task benchmark
- IFEval — size unspecified — instruction following benchmark
Baselines vs proposed
- Low-temperature sampling (LTS) baseline accuracy ranges used as zero-point; many methods produce higher log-probability sequences but not consistently higher accuracy against LTS (section 3.3).
- Beam search produces less probable and less accurate sequences compared to LTS baseline across datasets.
- Power sampling (SPS, power-SMC) reliably increases expected sequence log-probability but improvements in accuracy are dataset-dependent and inconsistent; on MATH500 power self-consistency improves over plain self-consistency, but on GPQA and MMLU it degrades performance.
- Tuning decoding method hyperparameters reliably increases log-probability within method but does not yield predictable correctness gains (Finding 2).
- Posttrained models consistently achieve more positive correlation coefficients between log-probability and correctness than base models across benchmarks.
Limitations
- Correlation analyses rely on existing datasets which may not cover all task types or reflect real-world diversity, possibly limiting generalization.
- Some decoded outputs are limited by max trajectory lengths (3072 tokens) due to computational constraints, potentially truncating longer reasoning chains.
- Within-sample correlation analysis is limited to low-temperature sampling with α=4 for repeated samples; effects may differ with other decoding setups.
- Lack of explicit adversarial or robustness evaluation — models tested under standard conditions without evaluating maleficent manipulations or distribution shifts.
- Some models (Qwen and Olmo3 families) and datasets are not fully open source, limiting reproducibility and external verification of results.
- Analysis focuses mostly on correlation metrics; causality or underlying mechanisms explaining deviations between probability and correctness are not deeply explored.
Open questions / follow-ons
- What causes the divergence between maximizing sequence probability and correctness when adjusting decoding hyperparameters or methods? Are there systematic biases in LLM probability distributions?
- Can improved calibration or training objectives that better align sequence probability with correctness enhance the efficacy of global decoding methods like beam search?
- How do these probability-correctness relationships hold under adversarial prompts or distribution shifts beyond standard benchmarks?
- Could combining sequence probability with other confidence metrics (e.g., model uncertainty, verifier models) produce more reliable correctness indicators at the sample level?
Why it matters for bot defense
For bot-defense and CAPTCHA practitioners relying on language models to generate or verify answers, this study highlights the complexities of using an LLM's own probability scores as a confidence or correctness signal. While higher sequence probability is somewhat predictive of correctness across datasets and models, relying on it at the decoding level or to adjudicate multiple answers from the same prompt is unreliable. This cautions against simplistic verifier-free approaches that trust the model's internal probabilities alone to improve the quality or correctness of generated responses. For CAPTCHA or bot-detection tasks involving natural language challenges, practitioners should combine probability with other evaluation signals and carefully tune decoding methods to avoid false confidence. Moreover, the observed dataset-dependent and model-specific variance suggests domain-specific validation is critical. Finally, power sampling and self-consistency strategies drawing on sequence probability should be applied with an understanding of their limitations, especially on tasks or prompts where the model's base accuracy is low.
Cite
@article{arxiv2606_27359,
title={ When are likely answers right? On Sequence Probability and Correctness in LLMs },
author={ Johannes Zenn and Jonas Geiping },
journal={arXiv preprint arXiv:2606.27359},
year={ 2026 },
url={https://arxiv.org/abs/2606.27359}
}