When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models
Source: arXiv:2606.27288 · Published 2026-06-25 · By Josef Chen
TL;DR
This paper investigates when combining multiple large language models (LLMs) via routing, voting, cascades, fusion, or mixture-of-agents (MoA) can improve accuracy beyond using a single model. The authors identify a fundamental performance ceiling: for any selection policy outputting one model's answer, the maximum ensemble accuracy cannot exceed 1 minus beta (β), where β is the probability that all models simultaneously fail on the same query. Common practice reports the average pairwise error correlation (ρ) as a diversity diagnostic, but this measure provably cannot identify β. Empirically across 67 state-of-the-art models from 21 providers, the authors show that error co-failure (β) is significantly underpredicted by Gaussian copula models calibrated to pairwise correlations, due to a common-mode failure atom beyond pairwise dependencies. This co-failure effect caps the achievable gain from combining models. They demonstrate two distinct regimes: on open-ended tasks like math and code, co-failure rates are substantial, and so the ensemble ceiling is limiting; whereas on multiple-choice science tasks, co-failure is negligible, and ensemble gains arise from resolvable disagreements among models. Finally, they show that learned routers are unable to realize nearly any oracle gain because query-level signals are too weak to identify the single best model for each question. Gains come primarily from models failing on different questions, not from simply adding more models or attempting complex voting/fusion.
Key findings
- The maximum accuracy of any policy outputting one model's answer is bounded by 1 - β, where β is the all-models-wrong failure rate on the same query (Prop. 1).
- Pairwise error correlation ρ cannot identify β; error laws with identical marginal error rates and pairwise correlations can have very different co-failure rates β (Prop. 3).
- A single-factor Gaussian copula calibrated on tetrachoric correlations underestimates empirical β by approximately 2.5× on a 67-model open-ended math benchmark MATH-500 (observed β=0.052 vs predicted β=0.021; 90% CI 1.7–3.4× range).
- The residual co-failure beyond Gaussian copula models originates from a common-mode failure atom not captured by pairwise dependencies (Prop. 2).
- On open-ended tasks like math and execution-graded code, β is significant (0.044–0.079) and bounds ensemble gains; on multiple-choice science tasks (GPQA-Diamond), β ≈ 0, opening more room for routing gain.
- Changing the answer format from multiple-choice to free-response on the same GPQA questions increases β from near zero to 0.127 with strong annotator agreement (κ=0.73–0.92), indicating co-failure depends on answer format, not content.
- Learned routing models recover near zero fraction of oracle gain G despite using strong features and large pools, indicating query-level routing signals are weak or absent.
- At matched average pairwise error correlation, low-ρ heterogeneous model ensembles outperform high-ρ self-mixture-of-agents ensembles, confirming that diversity helps only when members fail on different queries.
Threat model
This is an empirical and theoretical analysis paper on optimizing multi-model LLM ensembles for highest accuracy, not an adversarial security paper. The 'adversary' is effectively the natural joint error distribution of the model pool—how often all models err simultaneously—constraining how well any routing or ensemble scheme can perform. There is no active malicious attacker or adversarial input assumed. The limitations come from intrinsic model failures and their statistical dependence, not from data poisoning or evasion attacks.
Methodology — deep read
The study centers on the threat model of analyzing multi-model LLM orchestration policies—routing, voting, cascading, and mixture-of-agents—that output exactly one member model's answer. The adversary is essentially the natural error co-failure phenomenon in model pools (no active adversarial attacker). Key quantity β is defined as the probability that all models are jointly wrong on a query, a higher-order error dependence unidentifiable from pairwise correlations alone.
The data includes a market-scale collection of 67 models from 21 providers (GPT-5.5, Claude Opus 4.8, Gemini 3.1 Pro, Grok-4.3, and many others, including open-weight models) with live pricing information. Five main benchmarks cover open-ended math (MATH-500 and MATH-Hard), execution-graded code contests, and science multiple-choice surveys (GPQA-Diamond). Query sets range from 100 to 330 examples per benchmark with exact-match grading rules (no black-box LLM judging except for free-response GPQA). Several smaller 15-model subpools from 9 companies were also used to train learned routers.
To model dependence in model errors, the authors first measure marginal error rates and pairwise error correlations (both naive Pearson and latent-variable tetrachoric correlations) between model correctness indicators (binary). They fit single-factor and full covariance Gaussian copula models to generate predicted tail co-failure rates β. The copulas are compared against empirical β estimates with exact Clopper-Pearson confidence intervals from observed all-wrong counts. They rule out tail-dependent Archimedean copulas like Clayton, showing residual co-failure remains.
For learning routers, they train TF-IDF+logistic classifiers and gradient-boosted correctness predictors on 15-model pools, comparing performance to oracles selecting the actual best model per query. Multiple router architectures including a prompt-based LLM router were evaluated.
Evaluation uses exact-match accuracy, oracle gain G (difference between oracle and single-best model), mean pairwise ρ, and co-failure β with confidence intervals. They examine how G, β, and ρ vary by task domain, model pool size, and answer format. Ablations include clustering models by families and subsets.
The authors further introduce a finite-sample $0 certificate using Clopper-Pearson lower confidence intervals for β, bounding the maximal possible gain any routing/voting policy can realize from a given query set before deployment.
Market-scale experiments involve thousands of model calls costing approximately $270-$560, with full logs frozen allowing exact reproducibility from published oracles and scripts, though some datasets and models are closed.
One concrete example end-to-end: On the MATH-500 dataset with 67 models, they observe 17 all-wrong queries out of 330 (β=0.052) compared to a Gaussian copula predicted β of 0.021, demonstrating a ~2.5x co-failure underpricing, limiting the maximum achievable ensemble accuracy to about 94.8%. Learned routers on a smaller 15-model subset capture <10% of oracle gain G.
This rigorous, multi-pronged methodology combining theory, statistics, large-scale data, and economic modelling supports the core claims.
Technical innovations
- Identification and formal proof that co-failure rate β, not pairwise error correlation ρ, bounds the maximum accuracy achievable by any selection-based LLM ensemble policy.
- Introduction of a finite-sample, labels-free Clopper-Pearson confidence bound providing a practical $0 certificate on the maximal achievable gain before training any router or ensemble.
- Empirical measurement of co-failure across a large diverse pool of 67 frontier LLMs, revealing significant underestimation of tail failure by Gaussian copula models and naive correlation statistics due to a common-mode failure atom.
- Demonstration that co-failure and ensemble gain regimes depend on answer format (open-ended vs multiple-choice) rather than subject, via controlled experiments on GPQA-Diamond.
- Formal economic framing of orchestration as a priced assignment problem with a single shadow price λB, capturing cost-vs-quality tradeoffs in ensembles.
Datasets
- MATH-500 — ~330 queries — open-ended mathematics benchmark
- MATH-Hard Level-5 — ~298 queries — open-ended, harder math benchmark
- AIME-2024/2025 — unknown size — advanced math competition problems, new release
- GPQA-Diamond — 79 questions multiple-choice/free-response — graduate-level science questions
- Execution-Graded Code — 63 hard programming problems — graded by execution correctness
Baselines vs proposed
- Single-best model (e.g., Opus 4.8) accuracy on saturated multi-domain mix: 0.923 vs Oracle routing: 0.967 (Oracle gain=0.044, 95% CI [0.027,0.062])
- Single-best model accuracy on hard MMLU-Pro: 0.850 vs Oracle routing: 0.970 (Oracle gain=0.120, 95% CI [0.075,0.155])
- Single-factor Gaussian copula predicted β on MATH-500: 0.021 vs empirical β: 0.052 (~2.5× underpricing)
- Learned-LM router accuracy (TF-IDF+logistic) on 15-model saturated mix: 0.906 vs single-best 0.901 (captures 9% of G, 95% CI spans 0)
- Heterogeneous low-ρ ensembles outperform high-ρ Self-MoA at matched quality; majority vote gain regresses negatively on ρ with accuracy controls
Limitations
- Co-failure estimates β are based on limited numbers of all-wrong events (e.g., k=17 for MATH-500), leading to wide Clopper-Pearson confidence intervals.
- Learned routing evaluations limited to 15-model subpools and lack end-to-end training on the full 67-model market pool.
- Some datasets and prompts are closed or partially unavailable, complicating exact independent reproduction beyond published analyses.
- The study focuses on deterministic policies that always choose a single member's answer; other combination methods like answer fusion or debate are not bounded similarly.
- Routing gains may be under-estimated due to using static features (e.g., TF-IDF, prompt capsules) rather than potentially more informative latent features.
- The co-failure phenomenon is characterized mainly for open-ended vs multiple-choice format regimes; other task types or adversarial settings remain untested.
Open questions / follow-ons
- How can routing or fusion policies exploit more informative query- or model-level features to approach the oracle gain in practice?
- Can alternative combination methods that do not output strictly one member answer, such as weighted fusion or debate, circumvent the co-failure ceiling?
- What is the behavior of the co-failure tail β under distribution shift, adversarial queries, or in multi-modal model pools?
- To what extent do architectural or training differences between models affect the common-mode failure atom contributing to co-failure?
Why it matters for bot defense
Bot-defense and CAPTCHA practitioners should note that the paper's main insight is the fundamental limitation on gains from combining multiple decision models when all models fail jointly on some inputs. In anti-bot or CAPTCHA systems that ensemble or vote across multiple language models, this limits achievable error reduction. Pairwise error correlation observed between models is insufficient to predict how often all models fail simultaneously, so relying on simple diversity metrics risks overestimating ensemble benefits.
Deployers must measure or estimate the all-models-wrong co-failure tail (β) on their workloads to realistically assess the maximum accuracy gain possible through routing or voting. The paper provides a practical confidence-bound method to certify an upper limit on gains even before implementing expensive routing layers. It also shows that diverse inexpensive models help only if they fail on different inputs rather than just adding similar models increases co-failure risk. Lastly, the inability of learned routers to recover oracle gains indicates that strong query-level signals to predict the best model may be rare, limiting practical improvements from dynamic routing in real bot-defense scenarios.
Cite
@article{arxiv2606_27288,
title={ When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models },
author={ Josef Chen },
journal={arXiv preprint arXiv:2606.27288},
year={ 2026 },
url={https://arxiv.org/abs/2606.27288}
}