Challenges in Evaluating Explanation Methods for Static and Evolving Data

Source: arXiv:2608.06351 · Published 2026-08-06 · By Jerzy Stefanowski

TL;DR

This paper critically examines the challenges in evaluating Explainable Artificial Intelligence (XAI) methods, particularly focusing on both static and evolving data environments. It highlights the persistent insufficiency in rigorous evaluation approaches for XAI despite the proliferation of explanation techniques. Through two detailed case studies—DetoxAI for bias detection and concept unlearning in image recognition, and a human-grounded survey comparing saliency map-based explanation methods—the paper illustrates practical difficulties in assessing explanation quality, fairness, and usability. Further, it explores the under-addressed problem of adapting and evaluating explanations in streaming data settings subject to concept drift, proposing prototype- and counterfactual-based approaches to characterize and interpret evolving model behaviors over time. Overall, the paper argues for a more multifaceted evaluation framework combining automatic metrics, human-centered studies, and temporal dynamics considerations to better capture the effectiveness and trustworthiness of explanations.

Key contributions include empirical insights from bias mitigation using saliency-based explanations in convolutional networks, well-designed human experiments comparing explanation formats, and initial methods to track explanation stability and drift impact over time. The results demonstrate trade-offs in explanation method preferences depending on user expertise and specific features, highlight limitations of commonly used fidelity-based metrics, and underscore the challenges of stale or misleading explanations in non-stationary environments. The paper calls for more interdisciplinary and standardized evaluation protocols, richer benchmark datasets with ground truth explanation annotations, and deeper investigation of explanations' impact on human trust and decision-making in critical domains such as medical and aerospace applications.

Key findings

• In DetoxAI experiments on the CelebA face dataset, saliency maps revealed that 'wearing necktie' was a biased proxy for gender, causing misclassification of smiling status and lowering fairness metrics like Equalized Odds and Demographic Parity.
• Post-hoc concept unlearning methods (e.g., LEACE, AClarc) reduced unintended model bias, improving group fairness metrics and decreasing saliency focus on protected attributes, confirmed both qualitatively and quantitatively (Sec 3.1).
• Human-grounded evaluation with 148 participants showed ProtoPNet explanations were overall preferred (1,045 ranking points) over ACE (763) and RISE (643), indicating differing user preferences by explanation method and user XAI experience.
• Strong correlations were found between specific ProtoPNet or ACE explanation components and key anatomical features users expected for animal recognition (e.g., elephant trunk with ACE, tiger stripes with ProtoPNet), validating human relevance (Sec 3.2).
• Counterfactual explanation methods exhibit contradictory evaluation metrics (proximity vs sparsity vs plausibility), making single best method selection infeasible; ensemble-based multi-criteria approaches improve diversity and user-aligned trade-offs (Sec 4).
• Prototype and group counterfactual explanation sets adapted for evolving data streams can detect and characterize concept drift by tracking prototype centroid movement and cosine similarity of counterfactual vectors before/after drift (Sec 5).
• Static XAI evaluation measures focus on snapshot properties like fidelity and stability but fail to capture temporal dynamics and co-evolution of data, models, and explanations, highlighting the need for time-aware evaluation protocols.
• Human and application-grounded evaluations remain scarce (<5% of surveyed works), though they are essential to assess usefulness, trust, and human understanding of explanations beyond proxy metrics.

Threat model

The paper is not primarily a security-focused work with an adversarial threat model. Instead, the implicit adversary is any bias or opacity in AI models that harms fairness or user trust. The threat is that models encode and propagate unwanted correlations or that explanations become misleading due to stale information under evolving data. There are no active malicious attackers considered; rather, the concern is transparent, reliable explanations for human stakeholders.

Methodology — deep read

The paper's methodology is primarily analytical and experimental, comprising several components.

1. Threat Model & Assumptions: The work presumes adversaries or users as humans seeking explanations for black-box model decisions. The adversary model is not explicitly adversarial but rather focuses on fairness, bias detection, and interpretability for decision-makers. The assumption is that standard static XAI methods do not account for non-stationary data distributions encountered in dynamic environments.

2. Data: Two primary datasets are used in empirical studies. For bias analysis, the CelebA face image dataset with protected attribute labels (e.g., gender) is used; the model is a CNN trained on images without explicit features. For human-grounded studies, a subset of ImageNet classes featuring 10 well-known animal categories was selected for ease of human recognition and anatomical feature identification. Data preprocessing and CNN training details are described in the original master's thesis; exact dataset sizes and splits are not fully detailed but training achieves high accuracy.

3. Architecture and Algorithms: CNN architectures evaluated include multiple ResNet variants. Explanation methods studied include Concept-wise Relevance Propagation (CRP) for saliency maps detecting bias, and three explanation techniques for human survey: ACE (concept activation vectors), ProtoPNet (prototypical parts-based explanations), and RISE (randomized input sampling). Concept unlearning post-hoc methods—LEACE, AClarc, and threshold optimization—are applied to mitigate model bias.

For evolving data scenarios, prototype-based explanations for tree ensembles (bagging, random forests) and group counterfactual explanations (GCEs) are employed to analyze concept drift. Novel evaluation metrics for prototype displacement and cluster centroid shifts (e.g., mean minimal distance, cosine similarity) are introduced.

4. Training Regime: CNN models were trained on selected animal subsets until high accuracy; further details such as epochs or hardware are not explicitly given. For counterfactual ensembles, multi-criteria optimization uses Pareto fronts and Ideal Point Method without user preference elicitation.

5. Evaluation Protocol: Quantitative evaluation of bias uses group fairness metrics such as Equalized Odds and Demographic Parity; saliency maps are visually inspected and scored using metrics developed in prior work. Human evaluation involves an online survey with 148 respondents who rank explanation visualizations without knowledge of method names and complete demographic and expertise questionnaires. Correlation with expected anatomical features is computed. For evolving data, synthetic and real data streams test prototype and counterfactual drift metrics, cross-validated over intervals before and after detected drift events.

6. Reproducibility: Code and data availability are not explicitly mentioned, nor are frozen model weights or closed datasets indicated. Some referenced methods (e.g., CRP, GLANCE) are open-source per original publications, but overall reproducibility is partly dependent on external works.

Example: In the human study, 10 common animals from ImageNet were chosen, and CNNs trained for classification. Each animal was shown with explanation visualizations produced by ACE, ProtoPNet, and RISE. Participants rated explanations in an online survey. The results showed ProtoPNet scored highest overall, with preferences correlating to known anatomical features (e.g., stripes on tigers). This approach highlights complexities in standardizing explanations and the need for carefully designed interfaces and pilot studies to ensure meaningful human feedback.

Technical innovations

• Use of Concept-wise Relevance Propagation (CRP) saliency maps to detect biases encoded as proxy features in CNNs and to quantify unintended correlations with protected attributes.
• Application and comparison of multiple post-hoc concept unlearning methods to mitigate bias without full retraining, validated both visually and through group fairness metrics.
• Design and execution of a human-grounded evaluation protocol comparing multiple saliency- and prototype-based explanation methods with large diverse participant sampling, including standardized visualization.
• Introduction of new prototype drift metrics (e.g., mean minimal distance, centroid displacement) and group counterfactual evolution analysis (cosine similarity of explanation vectors) to characterize concept drift in evolving data streams.
• Proposal of a multi-criteria Pareto-front ensemble approach for counterfactual explanation selection that balances diverse and often conflicting evaluation metrics without user preference elicitation.

Datasets

• CelebA face images — ~200,000 images — publicly available
• ImageNet animal subset — hundreds to thousands of images per selected category — public, filtered subset

Baselines vs proposed

• ProtoPNet: human ranking points = 1045 vs ACE: 763 vs RISE: 643 in image recognition explanation study
• Concept unlearning methods vs baseline CNN on CelebA: group fairness metrics (Equalized Odds, Demographic Parity) improved after debiasing
• Single counterfactual explanation methods vs multi-criteria Pareto-front ensemble: ensemble produced explanations competitive or better on multiple criteria (no exact numeric values reported)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2608.06351.

Fig 1: The left panel shows raw face images, and the right panel, corresponding CRP

Limitations

• Evaluation results rely on specific image datasets (CelebA and ImageNet animals) limiting generalization to other domains or data types like text or time series.
• Human evaluation participants were mostly students and research staff, which may not represent lay or domain expert populations fully.
• Human studies focus on preference and feature correlation, but do not extensively measure trust, decision impact, or long-term user comprehension.
• Prototype and counterfactual drift metrics are preliminary; their effectiveness in diverse real-world streaming scenarios requires broader validation.
• No standardized, widely accepted ground-truth explanation datasets exist, complicating objective benchmarking and reproducibility.
• Codebase, exact model weights, and detailed training hyperparameters are not fully disclosed in the paper, which may hinder full replication.

Open questions / follow-ons

• How can XAI evaluation measures be standardized across explanation types, data modalities, and application contexts to enable fair comparison?
• What are the best practices for designing human-grounded evaluation studies that accurately capture user understanding, trust, and decision support impact?
• How to effectively integrate causal inference and background knowledge into explanations to improve their correctness and applicability?
• What dynamic explanation frameworks and measures can track co-evolution of data, models, and explanations in real-time non-stationary environments?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this paper underscores the complexity of evaluating explanation methods that might be integrated to understand or audit automated systems, including those detecting bots or adversarial behavior. It highlights that relying solely on proxy, fidelity-based metrics for explanations can be misleading, especially in dynamic environments where data and attacker strategies evolve. The human-grounded study insights imply that explanation methods should be tested with realistic users to ensure the explanations truly support their intended tasks, a critical factor when explanations inform security decisions or mitigate biases.

Furthermore, the exploration of evolving data and concept drift is especially relevant to bot detection as attacker tactics continuously change. Methods to track explanation stability and drift could inform adaptive bot detection mechanisms and maintain interpretability over time in live operational settings. Overall, bot-defense engineers should approach integrating XAI tools with caution, emphasizing thorough human-centered evaluation and monitoring temporal dynamics instead of assuming static model explanations remain valid.

Cite

@article{arxiv2608_06351,
  title={ Challenges in Evaluating Explanation Methods for Static and Evolving Data },
  author={ Jerzy Stefanowski },
  journal={arXiv preprint arXiv:2608.06351},
  year={ 2026 },
  url={https://arxiv.org/abs/2608.06351}
}

Read the full paper

• arXiv:2608.06351 (abstract)
• arXiv:2608.06351 (PDF)