The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs
Source: arXiv:2607.08734 · Published 2026-07-09 · By Baha Rababah, Cuneyt Gurcan Akcora, Carson K. Leung
TL;DR
This paper investigates the impact of post-training quantization (PTQ) on large language models (LLMs), focusing on the hidden assumption that standard metrics like accuracy and perplexity fully reflect behavioral equivalence between original and quantized models. Instead, the authors reveal an "illusion of equivalency": even when aggregate metrics remain stable, decision-level behaviors diverge significantly. They introduce correctness agreement, a new metric quantifying overlap in correct predictions between base and quantized models, independent of absolute accuracy. Empirical studies across multiple LLM architectures and quantization schemes ranging from 8-bit to 2-bit show that substantial behavioral drift emerges with increasing quantization, especially below 4 bits, despite stable or sometimes improved perplexity/accuracy.
To explain these findings, the authors perform a detailed statistical and divergence-based characterization of attention projection weights (query, key, value, output) at layer and block granularity. They find non-linear breakpoints where internal weight distributions and geometry drastically change at low-bit quantization (Q3 K and Q2 K). Query and key projections are more sensitive to quantization, exhibiting larger skewness, mean shifts, kurtosis collapse, and divergence, while value and output projections are more robust. This layered statistical sensitivity correlates with observed behavioral degradation. Their work highlights the limitations of relying solely on traditional metrics and motivates incorporating behavioral and internal structural evaluations for quantized LLMs.
Key findings
- Correctness agreement (decision-level overlap in correct predictions) is consistently lower than accuracy and decreases with lower bit-width, indicating behavioral drift even when accuracy remains near baseline (Table 4).
- Statistical metrics (mean, skewness, kurtosis) of attention projection weights remain stable for quantization ≥5 bits but show large distortions at 3-bit and 2-bit K-quantization, especially in query and key layers (Figs. 1, 2, 3).
- Divergence metrics (KL divergence, KS statistic) increase sharply at low bit-widths (Q3 K, Q2 K), confirming structural disruption in weight distributions (Fig. 4).
- Perplexity does not reliably correlate with behavioral consistency; some low-bit quantized models show perplexity similar or better than base models but have low correctness agreement (Tables 2, 3, discussion in 4.3).
- Query and key projection matrices exhibit higher sensitivity to quantization-induced distortion than value and output projections across all tested LLM architectures (Fig. 5, various per-layer analyses).
- Mid-range quantization levels (Q6 K to Q5 0) preserve statistical structure in weights and maintain behavior close to base models, balancing compression and fidelity.
- Legacy uniform quantization methods produce minimal structural weight distortions compared to statistically informed K-quantization, particularly at low bits.
- Model size affects sensitivity: smaller models (e.g., Llama-3.2-3B) show larger statistical deviations and behavioral degradation under aggressive quantization than larger ones (e.g., Vicuna 7B).
Threat model
N/A; this is not a security-focused paper but an empirical study of quantization effects on model behavior and weight distributions.
Methodology — deep read
The authors model post-training quantization (PTQ) as an operator family Tc acting on pretrained model parameters θ, producing quantized/dequantized parameters Tc(θ). They study multiple quantization configurations C spanning bit-widths from 8-bit down to 2-bit, including legacy uniform block quantization and statistically informed K-quantization schemes implemented in llama.cpp. The analysis focuses on self-attention projection matrices in transformer blocks (query Q, key K, value V, output O).
Data used includes open datasets: WikiText-2 and C4 for language modeling perplexity evaluation, and zero-shot benchmarks for downstream tasks (HellaSwag, Winogrande, ARC). These datasets enable measuring aggregate performance and behavioral consistency.
For structural analysis, they compute statistical summaries (mean, standard deviation, skewness, kurtosis) on the entries of each projection matrix post-quantization and measure their drift (Δ) from the base model. They also compute distributional divergences between base and quantized weight histograms using Kolmogorov–Smirnov statistic, Kullback–Leibler divergence, Euclidean distance, and cosine similarity. These stats and divergences are aggregated per layer and across all blocks.
For behavioral evaluation, they define correctness agreement (CA) as the fraction of examples correctly predicted by both base and quantized models, capturing decision-level consistency beyond aggregate accuracy. Correctness labels per example are derived under deterministic scoring rules for multiple-choice benchmarks.
Experiments quantize Llama-3.2-3B, Vicuna-7B-v1.5, Mistral-7B, and Llama-3.1-8B using llama.cpp quantizers. Weight matrices are extracted after dequantization for analysis. They evaluate performance using perplexity and task accuracy with multiple seeds on 8 NVIDIA Tesla V100 GPUs.
Evaluation compares weight-level distortions, perplexity, accuracy, and correctness agreement across quantization configurations and models to characterize the transition from high-fidelity preservation to disruptive behavior at reduced bit widths. The study also isolates layer-wise sensitivities to guide mixed precision quantization.
Reproducibility is supported by using publicly available models and datasets alongside the open-source llama.cpp quantizer; partial implementation code is included in the submission.
Technical innovations
- Introduction of correctness agreement, a decision-level behavioral metric measuring overlap in correct predictions between base and quantized LLMs, independent of absolute accuracy.
- Formalizing post-training quantization as a linear operator family acting on LLM parameters, enabling structural characterization at the weight distribution level.
- Layer-wise statistical and divergence-based sensitivity analysis identifying query and key projection matrices as disproportionately sensitive to low-bit quantization, unlike value and output projections.
- Controlled experimental sweep from 8-bit to 2-bit quantization across multiple LLMs and schemes linking internal structural metrics to downstream behavioral consistency for the first time.
Datasets
- WikiText-2 — ~2 million tokens — public
- C4 — 750+ GB web crawl — public
- HellaSwag — ~70k multiple-choice questions — public
- Winogrande — ~40k commonsense reasoning problems — public
- ARC (AI2 Reasoning Challenge) — ~7.8k science questions — public
Baselines vs proposed
- Base model: accuracy baseline = 55.5% (Llama-3.2-3B) vs Q2 K quantized: accuracy = 48.7%, correctness agreement = 38.5% (Table 4)
- Base model perplexity on WikiText2 (Llama-3.2-3B): 2.300 vs Q3 K quantized: 1.967 (improved perplexity but degraded behavioral agreement) (Table 2)
- Legacy quantization Q8 0: statistical metrics near-base (stable) vs K quantization Q3 K: significant distortion with kurtosis collapse and skewness spikes (Figs. 1-3)
- Cosine similarity of attention weights: ~0.507 at high bit-widths vs decreasing sharply below 4-bit K quantization (Fig. 4)
Limitations
- Behavioral evaluation uses zero-shot benchmarks with deterministic scoring; generalization to other tasks or interactive generation is untested.
- The analysis focuses on model weights and prediction correctness, omitting evaluation of other behavioral aspects like factual consistency, robustness, or safety.
- Quantization applies only post-training; effects under quantization-aware fine-tuning or other compression methods are not explored.
- Only four LLM architectures in the 3B to 8B parameter range are studied; results may not generalize to very large (100B+) or smaller models.
- No adversarial or worst-case input analysis to probe the robustness of quantized models against crafted perturbations.
- The correctness agreement metric depends on deterministic scoring and multiple-choice formats, limiting applicability to open-ended generation tasks.
Open questions / follow-ons
- Can mixed-precision quantization targeting sensitive Q and K layers improve efficiency while maintaining behavioral fidelity?
- How do these structural and behavioral distortions translate to open-ended generative tasks and interactive deployments?
- Would quantization-aware training or fine-tuning mitigate the non-linear distortions observed at low bit-widths?
- Can alternative behavioral metrics complement correctness agreement to capture other important semantic and safety properties under quantization?
Why it matters for bot defense
Bot-defense and CAPTCHA engineers employing quantized LLMs for NLP or adversarial detection should note that traditional metrics like accuracy or perplexity can conceal significant behavioral drift in quantized models. This drift means even modest quantization can change model output patterns unpredictably, potentially affecting reliability in security-sensitive applications.
Incorporating correctness agreement or similar decision-level behavioral metrics would provide a more faithful measure of whether a quantized model maintains the base model's predictive consistency. Furthermore, recognizing that query and key attention projections are disproportionately sensitive might guide selective precision allocation to preserve model robustness. Overall, this analysis urges engineers not to rely solely on surface-level performance metrics but to assess per-example decision consistency, especially when compressing LLMs for bot-detection pipelines.
Cite
@article{arxiv2607_08734,
title={ The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs },
author={ Baha Rababah and Cuneyt Gurcan Akcora and Carson K. Leung },
journal={arXiv preprint arXiv:2607.08734},
year={ 2026 },
url={https://arxiv.org/abs/2607.08734}
}