Can We Trust Item Response Theory for AI Evaluation?

Source: arXiv:2607.15190 · Published 2026-07-16 · By Han Jiang, Sunbeom Kwon, Jinwen Luo, Ziang Xiao, Susu Zhang

TL;DR

This paper critically examines the prevalent use of Item Response Theory (IRT) for AI model evaluation, specifically in the context of large language model (LLM) benchmarks. Traditional IRT methods were developed for human testing scenarios characterized by large numbers of examinees and relatively few items, with assumed normally distributed abilities. In contrast, AI benchmarks typically have far fewer models, vastly more items, and non-normal (skewed, multimodal) ability distributions. The authors conduct a large-scale, simulation-based empirical study to assess how these regime mismatches affect the computational feasibility, parameter recovery, and downstream inferences (model ranking, item diagnostics) of common IRT estimators (MML-EM, MCMC, VI, and a neural estimator). Drawing on real LLM benchmark data from six datasets and 18,000 simulation conditions, they reveal that classical estimators often fail to scale or converge on large benchmarks, while scalable methods sometimes yield unreliable item-level estimates or rankings in small or skewed model pools. The paper proposes concrete sample size guidelines and cautions on interpreting IRT-based claims in AI evaluation under these conditions.

Key findings

• MML-EM estimator failed to converge or ran out of memory in 69.45% of replications, especially on large benchmarks with >5,000 items (HELLASWAG, MMLU).
• MCMC provided most consistent parameter recovery but was computationally infeasible for benchmarks with large sample sizes (N=1000) or complex 3PL models (exceeding 72 hour runtime limits).
• VI estimator had a 10.71% overall failure rate, rising to 27.56% for 3PL models, and produced unreliable item difficulty estimates (< 0.5 correlation with ground truth for N ≤180 in some benchmarks).
• PSN neural estimator had 0% failure rate, scaled computationally better than MCMC and MML-EM but showed slightly less accurate and more variable parameter recovery than MCMC.
• Sample size critically affects item parameter recovery: N=30 models insufficient for reliable item difficulty and discrimination estimates (correlations below 0.5), while N≥100 substantially improved recovery.
• Model ranking recovery measured by Kendall’s τ degraded sharply as capability distribution skewness increased (from >0.85 for |skewness| <0.5 to below 0.60 for |skewness| >2.0), roughly independent of estimator choice.
• IRT-based benchmark compression using estimated Fisher information consistently outperformed random item subsets but only modestly; short-form ranking recovery was insensitive to sample size and estimator choice.
• Aggregate score error (difference between predicted and true model scores) was generally low (<0.025) under 2PL, but some estimator-model-size combinations showed elevated error and variance.

Threat model

The adversary is the AI evaluator using IRT-based statistical models to infer latent model abilities and item parameters from binary correctness responses. They are limited to observed response data and rely on estimation algorithms that assume normality and sufficient sample sizes. They cannot directly observe true latent parameters or perfectly correct for skewed/multimodal ability distributions in small model pools, risking biased or unstable estimates.

Methodology — deep read

The authors designed a large-scale simulation study grounded in representative AI benchmark data regimes, addressing three core IRT models (1PL, 2PL, 3PL) and four estimation methods recently used in AI evaluation:

1. Threat Model & Assumptions: The adversary is the benchmark evaluator relying on IRT to infer latent model capabilities and item characteristics from binary correctness responses without direct access to ground truth parameters. Assumptions challenged include normal ability distributions and favorable sample-to-parameter ratios, which are not met in AI benchmarking.

2. Data: They leveraged six public LLM benchmark datasets (ARC-CHALLENGE, HELLASWAG, MMLU, TRUTHFULQA, WINOGRANDE, GSM8K) from OpenLLM leaderboard scores, totaling 5,000–6,500 models per benchmark and 600–12,500 items each. After filtering out low-variance or saturated items and lowest performing models, the authors fit IRT models via variational inference to estimate 'true' item parameters and model capabilities.

3. Architecture/Algorithm: The study focused on unidimensional IRT logistic models — 1PL (Rasch), 2PL (adding discrimination), and 3PL (adding guessing). Four estimators were compared:

• Marginal Maximum Likelihood via EM (MML-EM) assuming normal ability priors,
• Bayesian Markov Chain Monte Carlo (MCMC) sampling,
• Variational Inference (VI) approximating posteriors,
• Neural Pseudo-Siamese Network estimator (PSN) which learns embeddings without parametric ability distribution assumptions.

4. Training Regime: For each of the 90 conditions (3 IRT models × 5 sample sizes N = 30, 100, 180, 400, 1000 × 6 benchmarks), 50 replicates were generated by sampling model abilities from empirical distributions and simulating binary responses. Estimators were run with standard settings in mirt, brms, py-irt packages, or the PSN implementation on a mix of CPU and GPUs.

5. Evaluation Protocol: They measured computational feasibility (failure rates, runtime), parameter recovery (Pearson correlations of estimated vs true item difficulty and discrimination, absolute errors for guessing parameter), latent ability ranking recovery (Kendall’s τ), aggregate score error, and short-form benchmark ranking recovery using Fisher information-based item selection. Multiple benchmarks and sample sizes allowed testing regime effects and estimator robustness.

6. Reproducibility: Code and exact implementation details appear documented, but the datasets rely on preprocessed proprietary leaderboard data; full replication may require access to these. They documented algorithmic configurations in appendices. Despite this, exact seeds or frozen weights were not explicitly stated.

One concrete example: For the MMLU benchmark with ~5,200 models and 12,500 items, the authors fit a 2PL model via variational inference to derive ground truth parameters. Then they simulated data with N=30 to 1000 sampled abilities and evaluated each estimator’s ability to recover these parameters and rankings. Results showed severe computational infeasibility for MML-EM at this large item scale, good ranking recovery at N≥100 under VI but poorer item parameter recovery at small N, and consistent performance degradation as ability skewness increased.

Technical innovations

• Systematic simulation study calibrating IRT evaluation to AI benchmarking regimes characterized by few models, many items, and non-normal ability distributions.
• Comprehensive comparison of four IRT estimators (MML-EM, MCMC, VI, PSN) on computational feasibility, parameter recovery, and inference validity under AI conditions.
• Empirical quantification of sample size and capability distribution skewness effects on reliability of IRT-derived model rankings and item diagnostics.
• Demonstration that neural-network based PSN estimator provides a scalable alternative to classical methods in large AI benchmarks without parametric ability assumptions.

Datasets

• ARC-CHALLENGE — ~5,221 models × 844 items — OpenLLM leaderboard
• HELLASWAG — ~5,221 models × 5,711 items — OpenLLM leaderboard
• MMLU — ~5,219 models × 12,508 items — OpenLLM leaderboard
• TRUTHFULQA — ~5,222 models × 644 items — OpenLLM leaderboard
• WINOGRANDE — ~6,550 models × 1,045 items — OpenLLM leaderboard
• GSM8K — ~6,068 models × 1,306 items — OpenLLM leaderboard

Baselines vs proposed

• MML-EM: failure rate = 69.45%, often infeasible on large benchmarks vs PSN: failure rate = 0%
• MCMC: most accurate parameter recovery but infeasible (>72 hrs) at N=1000 on large benchmarks vs VI: faster but less reliable item difficulty recovery (<0.5 correlation at N ≤180)
• VI: aggregate score error <0.025 under 2PL vs MML-EM under 1PL and VI under 3PL showing elevated error and variance
• Random benchmark subset: short-form ranking recovery Kendall’s τ ~0.5 vs IRT-based selection: consistently higher, ~0.6–0.7

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.15190.

Fig 6: Distributions of true capability (θ) and difficulty (b) parameters for TRUTHFULQA.

Fig 7: Distributions of true capability (θ) and difficulty (b) parameters for ARC-CHALLENGE.

Fig 8: Distributions of true capability (θ) and difficulty (b) parameters for WINOGRANDE.

Fig 9: Distributions of true capability (θ) and difficulty (b) parameters for GSM8K.

Fig 10: Distributions of true capability (θ) and difficulty (b) parameters for HELLASWAG.

Fig 11: Distributions of true capability (θ) and difficulty (b) parameters for MMLU.

Limitations

• Simulations restricted to unidimensional IRT models, excluding multidimensional or content-aware extensions.
• Hardware heterogeneity during computational benchmarks prevents precise infeasibility thresholds.
• No evaluation of algorithm convergence except for EM; other estimator convergence diagnostics underexplored.
• Sample sizes between 30 and 100 underexplored, limiting granularity of findings in this important transition range.
• Did not integrate item content or semantic features which could impact item quality beyond response patterns.
• Short-form benchmark effectiveness lacks clear factor attribution and warrants future explanation.

Open questions / follow-ons

• How do multidimensional IRT models and content-based item features affect parameter recovery and inference reliability in AI benchmarks?
• What are the convergence behaviors and robustness guarantees for neural-network based IRT estimators under varied AI data regimes?
• How does intermediate sample size (N between 30 and 100) specifically impact estimation stability and what diagnostics can preempt failure?
• What drives the modest gains in short-form benchmark effectiveness and can item selection criteria be optimized beyond Fisher information?

Why it matters for bot defense

Bot-defense and CAPTCHA engineers leveraging AI evaluation tools should treat IRT-derived metrics with caution under standard small-model, large-item conditions. The study highlights that classical psychometric IRT methods may fail to reliably rank models or identify informative challenge items when model populations are limited or capability distributions skewed. This implies that CAPTCHA challenge selection or bot detection benchmarks that rely on such latent trait modeling require careful sample size considerations and estimator choice, favoring Bayesian or neural methods with validated parameter recovery. Furthermore, the demonstrated feasibility constraints and propensity for distorted item parameter estimates underline the importance of simulation-based validation before deploying IRT-based metrics to drive security or usability decisions. Practitioners might also explore hybrid metrics combining IRT with heuristic or content-based item analysis to mitigate regime mismatch risks.

Cite

@article{arxiv2607_15190,
  title={ Can We Trust Item Response Theory for AI Evaluation? },
  author={ Han Jiang and Sunbeom Kwon and Jinwen Luo and Ziang Xiao and Susu Zhang },
  journal={arXiv preprint arXiv:2607.15190},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.15190}
}

Read the full paper

• arXiv:2607.15190 (abstract)
• arXiv:2607.15190 (PDF)