MixFrag: Fragility-Guided Mixed-Precision Post-Training Quantization for Vision Transformers

Source: arXiv:2607.28589 · Published 2026-07-30 · By Md. Mehrab Hossain Opi, Robiul Islam Ryad, Md. Umar Faruk

TL;DR

This paper addresses the problem of efficient post-training quantization (PTQ) for Vision Transformers (ViTs), which typically exhibit heterogeneous sensitivity to quantization across their components. Existing PTQ methods mostly use uniform bit-width quantization, failing to account for this fragility variation and thus leading to suboptimal precision allocation and accuracy drops at low bit-widths. The authors propose MixFrag, a novel mixed-precision PTQ framework that explicitly measures the quantization fragility of individual transformer components using KL divergence between full-precision and selectively quantized output distributions on a small calibration set. They then formulate the bit allocation as a Multiple-Choice Knapsack Problem (MCKP) and use dynamic programming to optimize the bit-width assignment under a global bit budget. Integrating this fragility-guided allocation with the AdaLog quantization backend produces mixed-precision ViTs with improved accuracy-compression tradeoffs.

Extensive evaluations on ImageNet-1K classification across seven ViT architectures demonstrate that MixFrag consistently outperforms uniform-precision PTQ for practical 4-6 bit settings, preserving classification accuracy without increasing average bit-width. More notably, on the COCO dataset for object detection and instance segmentation, MixFrag achieves state-of-the-art performance among mixed-precision PTQ methods, improving average precision (AP) by up to 9.6 points under stringent MP3/MP3 quantization constraints. The paper also provides a comprehensive analysis of the fragility metric, demonstrating its strong correlation with learned bit allocations and offering insights into layer-wise precision patterns. These results establish MixFrag as an effective and practical framework for fragility-aware mixed-precision post-training quantization of ViTs.

Key findings

• Fragility metric computed as KL divergence between full-precision and isolated quantized outputs effectively quantifies component-level quantization sensitivity.
• Dynamic programming solver for Multiple-Choice Knapsack Problem yields optimal per-layer bit allocation under average bit-width constraint.
• On ImageNet-1K, MixFrag improves top-1 accuracy compared to uniform-precision PTQ at 4-bit quantization across seven ViT backbones, closing accuracy gaps to full precision.
• On COCO object detection and instance segmentation tasks, MixFrag outperforms prior mixed-precision PTQ methods with up to 9.6 AP improvement under MP3/MP3 setting.
• Attention-related components (QKV, projection layers) receive higher bits in allocation, while MLP layers (FC1, FC2) are more aggressively quantized, reflecting fragility patterns.
• Fragility scores strongly correlate with final bit allocation, validating the metric's utility for guiding precision assignment.
• MixFrag's fragility-guided bit allocation leads to better downstream transfer performance on detection and segmentation than methods focused solely on ImageNet classification accuracy.

Threat model

The paper is not focused on a security threat model. The main adversary conceptually is resource constraints on hardware (memory, compute, and energy) limiting the representational precision of neural network parameters. The method assumes access to a pretrained full-precision Vision Transformer and a small calibration dataset but does not consider adversarial manipulation or model poisoning.

Methodology — deep read

1. Threat Model & Assumptions: The threat model is not security-oriented but focuses on deployment efficiency of pretrained Vision Transformers under quantization constraints. The adversary is the resource limitation—memory and compute budgets—not a malicious attacker.

2. Data: For calibration, MixFrag uses a small unlabeled calibration set—commonly 1,024 unlabeled ImageNet training images for classification tasks and 256 unlabeled COCO images for detection/segmentation. The fragility evaluation is conducted on these calibration samples without requiring labels or gradients.

3. Architecture/Algorithm: MixFrag targets quantizable components of ViTs, primarily linear projection layers in self-attention (QKV, output projection) and feed-forward networks (FC1, FC2). Operations like LayerNorm and residual connections remain in floating point. For each component c and candidate bit-width b (e.g., 2, 3, 4, 6 bits), the method quantizes c to b bits while keeping others full precision. It then computes the KL divergence between full-precision and partially quantized output logits (post-softmax with temperature T) over the calibration set, producing a fragility matrix Omega of size C components by B bit choices. This matrix reflects the expected performance degradation caused by each quantization option.

Next, they formulate mixed-precision bit allocation as a Multiple-Choice Knapsack Problem (MCKP): from each component's candidate bit-widths, select one assignment to minimize total expected fragility while satisfying a global average bit-width constraint. The MCKP is solved optimally using dynamic programming, efficiently navigating the combinatorial space.

The resulting layer-wise bit configuration is input to AdaLog, a state-of-the-art PTQ backend that performs adaptive logarithmic quantization per assigned bit-width, calibrates scale/zero-point parameters, and produces the final mixed-precision quantized model.

4. Training Regime: As post-training quantization, no retraining or gradient updates occur. Calibration uses only unlabeled data and forward passes. Experiments run mostly on a single NVIDIA RTX 4070 GPU.

5. Evaluation Protocol: Evaluation metrics include top-1 classification accuracy on ImageNet-1K validation set and average precision (AP) for bounding box and mask predictions on COCO validation. Experiments compare MixFrag to uniform-precision PTQ and existing mixed-precision PTQ methods like AdaLog, AQViT, LAMPQ, LRP-AQViT, and FIMA-Q. Ablations analyze bit allocations and fragility patterns. Statistical tests are not explicitly mentioned.

6. Reproducibility: The paper does not explicitly state code release status or frozen weights. Model provenance is from timm pretrained ViT, DeiT, and Swin backbones. Calibration datasets are subsets of public ImageNet and COCO.

Concrete Example: For DeiT-Tiny, the framework quantizes attention QKV layers more conservatively (higher bits) and quantizes MLP layers more aggressively under a 3-bit average bit budget. KL divergence fragility scores collected on 1,024 ImageNet images guide the MCKP solver to assign bits. AdaLog backend then applies corresponding quantization parameters individually. The final model maintains higher accuracy than uniform 3-bit quantization on ImageNet and transfers well on detection.

Technical innovations

• Introduction of a direct quantization fragility metric based on KL divergence between full-precision and selectively quantized outputs to precisely quantify component sensitivity.
• Formulation of bit-width allocation in mixed-precision PTQ for Vision Transformers as a Multiple-Choice Knapsack Problem solved optimally via dynamic programming to satisfy global bit budget constraints.
• Integration of fragility-guided dynamic bit allocation with AdaLog adaptive logarithmic quantization backend enabling flexible per-component precision assignment without retraining.
• Empirical demonstration that fragility-based bit allocation patterns correlate strongly with achieved model accuracy and outperform proxy importance measures used in prior mixed-precision methods.

Datasets

• ImageNet-1K — 1.28 million training images, 50,000 validation images — public dataset
• COCO — 118k training images, 5k validation images — public dataset

Baselines vs proposed

• AdaLog uniform 3-bit: average top-1 accuracy ~38.02% vs MixFrag mixed-precision 3-bit: 42.34%
• FIMA-Q uniform 3-bit: highest accuracy (~71.29% avg) on some ViTs vs MixFrag slightly lower but competitive on classification
• LRP-AQViT mixed-precision on COCO MP3/MP3: 31.3 AP vs MixFrag: 40.9 AP (+9.6 AP)
• LRP-AQViT mixed precision on COCO MP4/MP4: 45.0 AP vs MixFrag: 45.2 AP (+0.2 AP)
• Uniform 4-bit AdaLog on ImageNet: 75.61% average vs MixFrag 4-bit: 75.64%

Limitations

• MixFrag shows a larger accuracy drop at extremely aggressive 3-bit quantization on ImageNet classification compared to some costly optimization methods like FIMA-Q.
• The fragility estimation requires multiple forward passes per component and bit-width candidate, which may be computationally expensive for very large models or many candidates.
• Evaluation focuses primarily on vision tasks; the approach may need adjustment for other transformer applications like NLP.
• No adversarial or robustness evaluations under malicious perturbations are conducted, only deployment-related constraints are considered.
• Limited public codebase or pretrained weight releases may affect reproducibility.
• Statistical significance testing of results and run-to-run variance estimates are not reported.

Open questions / follow-ons

• Can the fragility metric be efficiently approximated to reduce compute overhead in large models or numerous bit candidates?
• How does fragility-guided mixed-precision quantization behave under distribution shift or domain adaptation scenarios beyond ImageNet/COCO?
• What are the theoretical guarantees linking KL divergence fragility with downstream fine-tuning or transfer learning robustness?
• Could the approach be extended or adapted for other transformer modalities such as NLP or audio transformers?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, MixFrag offers a systematic framework to efficiently quantize Vision Transformer models commonly emerging in vision-based security systems, including image-based bot detection or CAPTCHA analysis. The fragility-guided mixed-precision quantization approach can help deploy resource-intensive ViTs on constrained edge devices or embedded systems that run real-time bot classification or CAPTCHA verification, striking a better balance between latency, model size, and accuracy. Understanding the component-wise sensitivity to quantization also informs defensive model hardening strategies, as more fragile layers can be retained at higher precision for robustness. Although the paper focuses on image classification and object detection, its principles are broadly applicable to vision tasks in security contexts, where quantization-induced performance degradation can impact threat detection reliability. Captcha practitioners interested in deploying ViTs in low-power environments should consider integrating such fragility-aware quantization frameworks to optimize model efficiency without sacrificing effectiveness.

Cite

@article{arxiv2607_28589,
  title={ MixFrag: Fragility-Guided Mixed-Precision Post-Training Quantization for Vision Transformers },
  author={ Md. Mehrab Hossain Opi and Robiul Islam Ryad and Md. Umar Faruk },
  journal={arXiv preprint arXiv:2607.28589},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.28589}
}

Read the full paper

• arXiv:2607.28589 (abstract)
• arXiv:2607.28589 (PDF)