The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems
Source: arXiv:2607.19292 · Published 2026-07-21 · By Gjergji Kasneci, Enkelejda Kasneci
TL;DR
This perspective paper critiques the prevailing AI safety focus on visible, localized failures such as glaring harms, adversarial examples, or hypothetical catastrophes, arguing that this approach misses a critical class of quieter, systemic risks in modern AI deployments. These hidden failures are plausible, distributed across socio-technical layers, temporally extended across interactions and updates, and crucially degrade the human and organizational capacities to detect, contest, and recover from errors. The authors propose a novel five-layer framework for diagnosing these systemic integrity risks in modern AI systems: epistemic integrity (honest evidence and uncertainty representation), control integrity (robustness of authority and permissions), temporal integrity (safety continuity across sessions and drift), organizational integrity (institutional audit and intervention capacity), and ecosystem integrity (preservation of a healthy information environment). Within these layers, they identify emerging risk patterns such as overreliance, prompt injection, reward hacking, memory poisoning, evaluation deception, fictional oversight, synthetic evidence pollution, and model collapse.
Instead of developing new empirical benchmarks, the paper synthesizes existing literature and operationalizes broader system safety concepts to shift AI safety discourse beyond model-centric metrics towards socio-technical reliability. It offers several concrete design and governance recommendations to ensure AI deployments preserve conditions allowing error visibility, accountability, and recovery. This work situates AI safety within a systems engineering and organizational context, emphasizing that hidden integrity failures can make otherwise high-performing AI systems paradoxically more dangerous by eroding human and institutional safeguards over time.
Key findings
- Overreliance on AI outputs increases observed trust faster than warranted trust, creating calibration debt that is difficult to detect until severe errors occur (Fig. 2).
- Retrieval-augmented generation reduces hallucinations but can increase misplaced confidence through legitimacy laundering, where weak or contradictory evidence is presented as authoritative.
- Prompt injection attacks exploit systems' failure to separate instructions from data, threatening control integrity by collapsing authority hierarchies.
- Multi-turn, stateful AI interactions show safety drift, where risk accumulates across turns and sessions, escaping detection in single-turn snapshot evaluations (Fig. 3).
- Persistent memory modules in AI agents create novel safety risks including poisoning, privacy leakage, and silent propagation of adversarial content across sessions.
- Human-in-the-loop oversight often masks organizational integrity failures because humans lack real authority, time, or evidence to meaningfully intervene (termed fictional oversight).
- Recursive AI training on synthetic data leads to model collapse and ecosystem erosion, decreasing diversity and degrading the information environment relied upon for future oversight.
- Benchmark-centric evaluation fails to capture many operational, temporal, and organizational risks, enabling evaluation deception where high scores mask real-world failure modes.
Threat model
Adversaries include malicious actors exploiting AI model and system vulnerabilities (e.g., prompt injection, memory poisoning), but also broader emergent threats from complex socio-technical interactions such as organizational misuse, cumulative multi-turn drift, and ecosystem degradation. The adversary can manipulate information, permissions, or institutional processes but is assumed not to have unlimited ability to rewrite all deployed components or instantly detect system internals.
Methodology — deep read
As a conceptual perspective, the paper does not present new empirical experiments but synthesizes a broad range of recent theoretical and empirical AI safety literature through a systems safety lens.
Threat Model & Assumptions: The adversary is modeled not only as an attacker exploiting specific AI model vulnerabilities (e.g., prompt injections) but also includes emergent socio-technical threats arising from compositional interactions within deployed AI ecosystems. The paper assumes adversaries may be human attackers, organizational actors, or systemic processes (e.g., automation bias) capable of inducing subtle, long-term failures that evade model-level detection.
Data: The paper surveys recent studies, benchmarks, red-teaming work, and incident reports spanning language models, retrieval systems, tool-using agents, and institutional AI deployments. Details on dataset sizes or labels are not specified since this is a conceptual synthesis.
Architecture / Algorithm: Rather than proposing a single technical solution, the authors introduce a five-layer diagnostic framework—epistemic, control, temporal, organizational, ecosystem integrity—to categorize and analyze hidden AI safety risks. This framework inputs various facets of modern AI systems (e.g., retrieval modules, memory, human oversight) with outputs being an integrated understanding of system-wide integrity failures.
Training Regime: Not applicable; however, the paper references relevant training phenomena like recursive model collapse arising from training on AI-generated data.
Evaluation Protocol: The authors critique standard single-turn benchmark testing and advocate for multi-turn, trajectory-level safety evaluations, human factors investigations, adversarial testing, and organizational audit assessments to capture complex emergent risks.
Reproducibility: No direct code or datasets are released as this is a perspective article. The cited studies may have associated public artifacts. The work primarily serves as an operational lens and agenda-setting synthesis rather than a reproducible experiment.
To concretize, consider epistemic integrity: a deployed retrieval-augmented LLM may present answers with linked passages. However, if it shows only selectively supportive or outdated sources without disclosing uncertainty, users can become over-reliant (calibration debt). Over time, this erodes user skepticism and institutional checking mechanisms, representing a hidden failure invisible in snapshot outputs but dangerous at system level.
Technical innovations
- Five-layer socio-technical integrity framework explicitly modeling epistemic, control, temporal, organizational, and ecosystem risks in AI safety.
- Identification and naming of 'calibration debt' as a socio-technical accumulation of miscalibrated user reliance divorced from warranted reliance.
- Highlighting prompt injection as a failure of instruction authority and calling for architectural separation and cryptographic attestation of instruction provenance.
- Emphasizing temporal integrity risks like safety drift and persistent memory poisoning that evade standard benchmarking.
- Conceptualizing ecosystem integrity to capture recursive training risks from synthetic content leading to model collapse and erosion of future oversight capability.
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.19292.
Fig 1: The Hidden Iceberg of AI Safety: From Model Outputs to Systemic Integrity.
Fig 2: Calibration debt in AI-assisted use. Repeated satisfactory interactions can increase
Fig 3: Conceptual illustration of temporally extended and ecosystem-level AI safety
Limitations
- No novel empirical data or quantitative prevalence estimates are provided; the framework synthesizes existing literature and perspectives.
- Absence of concrete quantitative validation or simulation of proposed integrity failures and mitigations.
- Limited detail on how to operationalize and measure many organizational and ecosystem-level risks in practice.
- The framework is high-level and conceptual, which may be difficult to apply directly without tailored context-specific methods.
- Potential gaps in coverage if future unforeseen socio-technical failure modes arise beyond the five layers.
- Not covering certain specific technical robustness or interpretability research in depth, as the focus is on systemic integrity.
Open questions / follow-ons
- How to rigorously measure and audit each of the five integrity layers in operational AI deployments?
- What technical mechanisms can best enforce instruction authority separation and cryptographic provenance guarantees at scale?
- How can long-term risk accumulation and safety drift be proactively detected and mitigated across multi-turn AI interactions?
- What governance models and organizational structures effectively preserve organizational integrity and prevent fictional oversight?
Why it matters for bot defense
Bot-defense and CAPTCHA practitioners can apply the five-layer integrity framework to better understand hidden systemic risks beyond individual model outputs relevant to automated adversaries. For example, epistemic integrity considerations highlight the risk that adversarial bots might exploit overreliance or legitimacy laundering in retrieval-augmented CAPTCHAs. Control integrity concerns around instruction authority emphasize the need to design CAPTCHA systems robust to prompt injection or content-based hijacking. Temporal integrity reminds that bot defenses must consider multi-step attack chains or evolving adversarial strategies over time, not just isolated challenges. Organizational integrity warnings encourage maintaining rigorous human auditing, review capacity, and incident reporting rather than relying solely on automated metrics. Ecosystem integrity points to the risk that widespread bot usage and synthetic content might degrade the trustworthiness of CAPTCHA training data or blacklists. Overall, this work encourages bot-defense engineers to move beyond narrow model metrics toward holistic socio-technical reliability frameworks encompassing user behavior, layered controls, temporal dynamics, and governance to maintain robust defenses against increasingly sophisticated automated adversaries.
Cite
@article{arxiv2607_19292,
title={ The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems },
author={ Gjergji Kasneci and Enkelejda Kasneci },
journal={arXiv preprint arXiv:2607.19292},
year={ 2026 },
url={https://arxiv.org/abs/2607.19292}
}