Plug-in Losses for Evidential Deep Learning: A Simplified Framework for Uncertainty Estimation that Includes the Softmax Classifier
Source: arXiv:2605.22746 · Published 2026-05-21 · By Berk Hayta, Hannah Laus, Simon Mittermaier, Felix Krahmer
TL;DR
This paper addresses a key challenge in Evidential Deep Learning (EDL), which aims to estimate uncertainty in classification tasks by modeling class probabilities as Dirichlet distributions parameterized by a neural network. While classical EDL provides single-pass uncertainty estimates, it requires optimizing complex expected Dirichlet losses that are computationally involved and can complicate training stability and convergence. The authors propose replacing the Dirichlet-expected losses with simpler plug-in losses evaluated at the Dirichlet mean, showing theoretically that the approximation error decays inversely with the total evidence. This framework also naturally includes standard softmax classifiers as a special case, connecting classical softmax with uncertainty estimation in the evidential sense.
Empirically, the authors validate the plug-in loss approximations on the Google Speech Commands v1 dataset, a realistic speech keyword spotting benchmark. They find that models trained with the simplified objectives achieve predictive accuracy and selective prediction performance comparable to classical EDL, but with simpler implementation via standard loss functions like cross-entropy and mean squared error. They also evaluate uncertainty scoring via entropy and vacuity and show that KL regularization impacts vacuity's efficacy for selective prediction. Overall, the work rigorously bridges theoretical justification and practical simplification of evidential uncertainty estimation in classification, especially for resource-constrained or real-time settings.
Key findings
- The plug-in loss approximation to the classical Dirichlet-expected losses has an error term R(α, y) = O((α0 + 1)^{-1}), where α0 is the total Dirichlet concentration (Theorem 2).
- Softmax classifiers are special cases of simplified evidential classifiers under the plug-in framework (Theorem 1).
- On Google Speech Commands v1, simplified plug-in loss models (Plug-in EDL-CE, Plug-in EDL-MSE) match classical EDL-CE and EDL-MSE in base classification accuracy (e.g., ~96.8% vs 96.9%) and selective prediction accuracy at 99.0%, 99.5%, 99.9% thresholded accuracies.
- Entropy-based selective prediction generally yields higher coverage-accuracy trade-offs than vacuity-based selection on keyword spotting.
- KL regularization notably improves vacuity-based selective prediction accuracy (e.g., Softmax+KL vacuity total accuracy at 99.9% threshold = 80.36% vs Softmax vanilla 62.00%).
- Models trained with plug-in losses are simpler to implement with standard cross-entropy or MSE losses and are compatible with existing training pipelines.
- Vacuity alone is insufficient as a reliable uncertainty score without KL regularization, while entropy is robust across variants.
- The paper provides the first empirical coverage-accuracy trade-offs for speech recognition tasks using evidential uncertainty estimation.
Threat model
Not applicable. The paper does not explicitly address adversarial or malicious threat actors but focuses on developing and simplifying uncertainty estimation methods within probabilistic classification frameworks.
Methodology — deep read
The paper focuses on supervised multiclass classification with K classes, where the model predicts logits z and converts these into Dirichlet parameters α to model uncertainty over class probabilities via a Dirichlet distribution.
Threat model & assumptions: The adversary or out-of-distribution scenario is not explicitly addressed as this is primarily an uncertainty estimation methodology. The model assumes data is i.i.d. and sets in-distribution evaluation via predictive uncertainty and selective prediction.
Data: They use the Google Speech Commands v1 dataset, a 30-class keyword spotting benchmark with standard train, validation, and test splits. Preprocessing and augmentation follow the NVIDIA NeMo MatchboxNet pipeline.
Architecture/algorithm: Models build upon the MatchboxNet architecture, a convolutional model optimized for efficiency and accuracy in keyword spotting. The neural network outputs logits z; these logits are mapped componentwise via a monotone function τ (such as softplus or exponential) to evidence e, then to Dirichlet parameters α=ϕ(e) (often αi=ei+1 classical EDL, or simply ei in plug-in cases). Predictions are probabilities derived as the Dirichlet mean α/α0. Classical EDL optimizes the expected loss over the Dirichlet (e.g., cross-entropy or MSE) plus an optional KL regularizer penalizing evidence on incorrect classes. The authors propose replacing the expected Dirichlet loss with a plug-in loss evaluated at the Dirichlet mean, simplifying the optimization.
Training regime: Common optimizer, learning rate schedule, and epochs were used across models for comparability (details in Appendix C.1). KL regularization schedules were adopted where relevant. Multiple runs (n=5) were performed for error bars.
Evaluation protocol: Predictive accuracy and selective prediction were measured by thresholding uncertainty scores (entropy or vacuity) to withhold uncertain predictions, following a coverage-accuracy trade-off framework. Thresholds were chosen to meet target accuracies (e.g., 99%, 99.5%, 99.9%) and total accuracy and coverage reported. Comparison was done across classical EDL losses and simplified plug-in losses, as well as standard softmax and variants with or without KL regularization.
Reproducibility: Code and detailed experiment protocols are referenced though not released at the time of writing due to typical research paper embargoes. Dataset is public.
End-to-end example: For a speech input x, the MatchboxNet outputs logits z. By applying the evidence map τ and the additive constant c (e.g., αi=ei+1), the Dirichlet α is computed. Instead of calculating the expected loss over Dirichlet samples, the plug-in loss ℓ(α/α0,y) is computed using standard cross-entropy or MSE. Gradients update θ to minimize empirical risk over training. During inference, predictive probabilities α/α0 provide classification and uncertainty scores (entropy or vacuity). Thresholding rejects predictions above uncertainty thresholds, trading off coverage for higher accuracy.
Technical innovations
- Novel theoretical proof that the classical EDL expected Dirichlet losses can be tightly approximated by plug-in losses evaluated at the Dirichlet mean, with an explicit remainder bound decaying inversely with evidence concentration.
- Unification of standard softmax classification as a special case of the evidential plug-in loss framework under a specific logits-to-Dirichlet mapping.
- Introduction of a simplified evidential classifier concept where training depends only on the predicted probabilities (Dirichlet mean) rather than the full Dirichlet expected loss.
- Empirical demonstration that plug-in losses retain predictive accuracy and operational selective prediction performance comparable to classical EDL but are simpler to implement and optimize.
Datasets
- Google Speech Commands v1 — ~105K audio samples — public dataset
Baselines vs proposed
- EDL-CE: Base accuracy = 96.88% ±0.37 vs Plug-in EDL-CE: 96.84% ±0.38
- EDL-CE total accuracy at 99.9% entropy threshold = 93.61% ±1.26 vs Plug-in EDL-CE = 93.39% ±1.57
- Softmax vacuity total accuracy at 99.9% = 62.00% ±17.09 vs Softmax + KL vacuity total accuracy = 80.36% ±7.60
- EDL-CE no KL vacuity total accuracy at 99.9% = 47.14% ±15.60 vs EDL-CE with KL = 81.62% ±5.81
- Softmax entropy total accuracy at 99.9% = 88.41% ±2.39 vs Plug-in EDL-CE entropy total accuracy = 93.39% ±1.57
Limitations
- Empirical evaluation limited to one task and dataset (Google Speech Commands v1) restricting generalization claims.
- No explicit evaluation on out-of-distribution data or adversarial robustness is provided.
- Selective prediction experiments focus on in-distribution uncertainty thresholding rather than deployment scenarios with distribution shift.
- Theoretical approximations assume high evidence (high concentration) regime; behavior with low evidence uncertain.
- Effect of hyperparameters for KL regularization and plug-in loss mapping require further systematic tuning and analysis.
- Uncertainty metrics explored are limited to entropy and vacuity; other measures like mutual information are not studied.
Open questions / follow-ons
- How does the plug-in approximation framework perform on other datasets, data modalities, and under distribution shift including out-of-distribution detection tasks?
- What is the behavior of uncertainty estimation with low evidence concentration settings where the theoretical bounds are weaker?
- Can vacuity and entropy uncertainty scores be combined with multi-threshold selective prediction strategies to improve deployment reliability?
- How does KL regularization interact with different plug-in loss variants and what is the ideal scheduling and weighting for operational uncertainty?
Why it matters for bot defense
This work offers bot-defense and CAPTCHA practitioners a simplified yet theoretically principled approach to incorporating uncertainty estimation using Evidential Deep Learning. The plug-in loss framework reduces implementation complexity while retaining key uncertainty quantification benefits, facilitating easier integration into existing models without bespoke loss functions. Since many bot-detection systems require reliable confidence estimation for selective gating (e.g., challenging users only on uncertain samples), simplified evidential classifiers could improve efficiency by enabling uncertainty-based selective prediction with a single forward pass. Importantly, the connection to standard softmax classifiers allows practitioners to interpret and leverage existing softmax networks within an evidential uncertainty framework without substantial retraining overhead. However, limitations remain regarding out-of-distribution scenarios and adversarial analysis, which are critical in bot-detection contexts; thus further evaluation is essential before deployment. Nevertheless, the ability to produce uncertainty scores like entropy or vacuity straightforwardly may enhance CAPTCHA decision rules that depend on predictive confidence or uncertainty thresholds.
Cite
@article{arxiv2605_22746,
title={ Plug-in Losses for Evidential Deep Learning: A Simplified Framework for Uncertainty Estimation that Includes the Softmax Classifier },
author={ Berk Hayta and Hannah Laus and Simon Mittermaier and Felix Krahmer },
journal={arXiv preprint arXiv:2605.22746},
year={ 2026 },
url={https://arxiv.org/abs/2605.22746}
}