Quantum Fidelity-per-Cost: A Metric for Evaluation of Quantum Computing Systems

Source: arXiv:2607.28572 · Published 2026-07-30 · By Siddarth Shinde, Jakub Szefer

TL;DR

This paper addresses the practical challenge of comparing cloud-accessible quantum computing systems by incorporating both execution fidelity and monetary cost into a single, cost-aware metric termed Quantum Fidelity-per-Cost (QFC). The authors perform an empirical cross-provider study of a simple benchmark workload—the preparation of a two-qubit Bell state—across 14 quantum processing unit (QPU) access-path entries representing 12 distinct physical devices from major vendors including AWS Braket, IBM Quantum Runtime, IQM Resonance, and Oxford Quantum Circuits (OQC). They aggregate 247 runs at shot counts ranging from 20 to 1000, capturing fidelity via Kullback–Leibler divergence relative to the ideal output distribution and measuring cost under varied cloud provider billing models. The main finding is that a cost-aware ranking of devices can significantly differ from a fidelity-only ranking; devices with the best raw fidelity are not necessarily the best value for users constrained by budget. Furthermore, the billing model itself governs how QFC scales with shot count, not the hardware quality directly. The authors provide a reproducible pipeline, openly released measurement data, and analyze metric robustness under parameter reweighting and alternative billing assumptions.

Key findings

• Collected 247 quantum circuit executions across 14 QPU access-paths (12 distinct physical QPUs) spanning 4 cloud providers with shot counts from 20 to 1000.
• Defined QFC metric combining KL divergence-based fidelity, shot count, and monetary cost with tunable parameters α, β, γ; using α = β = γ = 1 as default.
• Fidelity-only ranking at 1000 shots ranked IBM Marrakesh best with mean TVD 0.0254; QFC ranking placed IQM Sirius first with QFC 608.21 and IBM Marrakesh fifth.
• QFC values span approximately 1.5–2 orders of magnitude between best and worst devices at 1000 shots; top QFC device is roughly 47× better value than lowest.
• Shot-scaling of QFC is determined by billing model: per-shot-only costs yield flat QFC vs shots, task + per-shot models show saturating increase, and runtime-based costs produce near-linear increase (Fig 3).
• Access-path (software/cloud interface) to the same physical QPU can differ markedly in QFC due to different fidelities and costs; e.g., IQM Emerald on AWS vs IQM Resonance.
• Trapped-ion IonQ Aria-1 had best fidelity (mean TVD 0.0245) but lowest QFC due to high per-shot cost ($0.03 + $0.30 task fee), illustrating that hardware modality alone does not determine value.
• Ranking by QFC is robust to moderate changes in weighting parameters α, β, γ (Kendall τ ≥ 0.71), with IQM Sirius top-ranked in 9 out of 10 parameter scenarios.

Methodology — deep read

1. Threat Model & Assumptions: The paper does not focus on adversarial threats but considers a user selecting among cloud quantum computing providers with known billing models and noisy hardware. Assumes the user wants to maximize fidelity per budget under heterogeneous billing.

2. Data: The authors conducted 247 runs of a two-qubit Bell-state circuit—a minimal, controlled workload—across 14 QPU access-path entries representing 12 unique physical quantum processors from four cloud vendors (AWS Braket, IBM Quantum Runtime, IQM Resonance, OQC cloud). Shot counts per run varied in {20, 50, 100, 500, 1000}, although coverage was uneven (some missing 50-shot data). Measurement data captures raw counts per two-qubit outcome per run.

3. Architecture/Algorithm: They use Total Variation Distance (TVD) and Kullback–Leibler (KL) divergence to measure fidelity relative to the ideal Bell-state distribution (equal weight on |00⟩ and |11⟩ outcomes). To combine fidelity and cost, they define a Quantum Fidelity-per-Cost metric (QFC):

   QFC(N) = exp(-α * KL(P_emp || P_ideal)) * N^γ / C(N)^β

   where N is shot count, C(N) is run cost per provider billing model, and α, β, γ are sensitivity parameters (default 1). KL is computed with a smoothing constant to handle zero-probability outcomes.

4. Training Regime: Not applicable; this is an empirical benchmarking study without machine learning training.

5. Evaluation Protocol: Each run yields measurement counts from which TVD and KL divergences are computed. Costs are calculated via provider-specific billing formulas: either per-task + per-shot, per-shot only, or runtime-based billing. QFC is computed for each run and aggregated. Rankings by fidelity and QFC are compared at fixed shot budgets, showing differences. Sensitivity analyses vary α, β, γ and billing assumptions; Kendall rank correlation is used to measure ranking stability.

6. Reproducibility: Raw run data, cost parameters, and evaluation pipeline are publicly released on GitHub, enabling exact re-computation of all scores and figures without provider credentials or hardware re-execution. This supports reproducibility and updating rankings as pricing or hardware changes.

Example end-to-end: For a given run of the Bell state circuit on IBM Marrakesh at 1000 shots, the empirical output distribution P_emp is collected. KL divergence D_KL(P_emp||P_ideal) is computed against the ideal Bell state distribution. The cost C(N) at 1000 shots is calculated using IBM Runtime's per-second pricing and the measured runtime. Then QFC(1000) is computed as exp(-D_KL) * 1000^1 / C(1000)^1. This score is compared against other devices similarly measured to produce fidelity-only and cost-aware rankings.

Technical innovations

• Introduction of Quantum Fidelity-per-Cost (QFC), a concrete metric integrating fidelity (via KL divergence), shot count, and monetary cost into one score aligned with heterogeneous cloud quantum billing models.
• Cross-provider empirical measurement study spanning 14 cloud QPU access-path entries across 4 major vendors, enabling consistent cost-aware fidelity comparison under real-world pricing heterogeneity.
• Categorization of cloud quantum billing models into four practical classes and incorporation into unified cost terms, enabling transparent comparison despite runtime-, task+shot-, or per-shot pricing semantics.
• Demonstration that billing model, not hardware quality, determines scaling of QFC with shot count, underscoring that cost-aware rankings must be evaluated at fixed shot budgets.
• Open-source release of a reproducible benchmarking pipeline, run data, and cost models, allowing continuous updates with evolving quantum hardware and pricing.

Datasets

• Bell-state executions dataset — 247 runs — publicly released at https://github.com/caslab-code/qc-fidelity-and-cost

Baselines vs proposed

• Fidelity-only rank (lowest mean TVD @ 1000 shots) IBM Marrakesh: 0.0254 vs. QFC top device IQM Sirius: QFC = 608.21 (rank 1), IBM Marrakesh QFC = 217.93 (rank 5)
• Best fidelity device IonQ Aria-1 mean TVD: 0.0245 vs. QFC = 27.84 (rank 14 of 15, lowest value due to high cost)
• Per-shot only billing QFC growth (flat with shots) vs. task + per-shot billing (rises then saturates) vs. runtime billing (rises approx. linearly with shots) (Fig 3)
• Ranking stability under varied QFC weights α, β, γ yields Kendall τ from 0.56 to 1.0 with IQM Sirius top in 9/10 cases (Table III)

Limitations

• Workload limited to a single two-qubit Bell-state circuit; does not generalize to deeper, multi-qubit circuits or application-scale workloads.
• Snapshot in time: pricing, calibration data, and hardware used reflect a limited temporal window; results may change as new machines or prices evolve.
• Cost models rely on provider-reported runtime fields that differ in semantics and resolution, confounding direct runtime-cost comparison across vendors.
• No evaluation of queue/wait times, retry rates, or reliability costs, which could impact total cost and user experience.
• Variance in runtime billing due to infrastructure jitter is significant for some devices, causing large cost variation unrelated to fidelity.
• KL divergence choice and smoothing constant ϵ influence QFC values and ranking; alternative divergences might yield different results.
• Multiple access paths per physical QPU complicate attribution of performance differences purely to hardware.

Open questions / follow-ons

• How well does QFC extend to deeper and larger-scale quantum circuits beyond two-qubit Bell states, especially application-relevant workloads like QAOA or VQE?
• How to incorporate queueing delay, reliability, and retry costs into cost-aware fidelity metrics for a comprehensive economic comparison in real cloud usage?
• Can predictive models be developed to estimate QFC before execution, enabling cost-aware scheduling and backend selection dynamically?
• What is the impact of alternative fidelity divergence measures (e.g., Jensen–Shannon, Hellinger) or cost definitions on the stability and interpretability of QFC?

Why it matters for bot defense

While this research does not directly address bot defense or CAPTCHA challenges, its approach to integrating heterogeneous resource costs with performance metrics is instructive for CAPTCHA engineers who must balance usability, security, and computational cost in deployment. Analogously, cost-aware ranking of backend systems or service providers may help CAPTCHA platforms optimize challenge generation or verification choices under budget constraints. The paper’s emphasis on reproducible scoring and sensitivity to pricing modalities might inspire analogous cost-fidelity metrics in large-scale CAPTCHA deployment models, particularly when considering cloud-based solver infrastructure or adaptive challenge schemes.

Cite

@article{arxiv2607_28572,
  title={ Quantum Fidelity-per-Cost: A Metric for Evaluation of Quantum Computing Systems },
  author={ Siddarth Shinde and Jakub Szefer },
  journal={arXiv preprint arXiv:2607.28572},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.28572}
}

Read the full paper

• arXiv:2607.28572 (abstract)
• arXiv:2607.28572 (PDF)