The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems

1. Introduction: Beyond the “Shocking Output”

Current discourse in AI safety frequently prioritizes visible, localized malfunctions—what Kasneci et al. (2026) characterize as the “tip of the iceberg.” As illustrated in the authors’ framework (Figure 1), these visible failures include shocking model outputs, explicit policy violations, and vivid adversarial examples. While these issues are immediately legible, they are incomplete representations of the risks inherent in deployed systems.

The true safety-critical challenges are the submerged “hidden systemic failures.” These risks do not arise from isolated model-level errors but from the complex interactions within the broader deployment stack. This operational synthesis suggests that AI safety is an emergent property of the entire socio-technical system—encompassing models, retrieval layers, organizational oversight, and the information ecosystem—rather than a narrow, scalar property of a model’s parameters.

2. The Four Pillars of Silent Failure

The authors identify four shared features of under-recognized AI safety challenges that allow them to evade standard instrumentation and instrumentation:

• Plausibility: Failures often mimic ordinary, correct operations closely enough that users and organizations normalize them, overlooking hazards because they lack spectacular “brokenness.”
• Distribution: Errors are rarely localized to a single component. Instead, they emerge from the interplay between the model, the retrieval-augmented generation (RAG) pipeline, and human interfaces.
• Temporal Extension: Risk is not a snapshot; it is temporally extended, accumulating silently across multi-turn interactions, persistent memory updates, and long-term deployment drift.
• Correction Degradation: Crucially, these failures constitute a recursive failure. The system does not merely produce errors; it erodes the “epistemic infrastructure”—the human skepticism and institutional capacity—required to notice and correct those errors.

3. A Framework for Socio-Technical Integrity

To diagnose these submerged risks, the authors propose a five-layer framework of integrity layers.

3.1 Epistemic Integrity

This layer evaluates whether evidence and uncertainty are represented honestly enough to support calibrated human reliance.

• Calibration Debt: A growing mismatch where observed human reliance on a system increases through repeated use faster than the system’s actual reliability warrants.
• Uncertainty Laundering: The process by which weakly supported or speculative model outputs are converted into polished, authoritative artifacts that shed their original evidentiary gaps.

3.2 Control Integrity

Control integrity examines whether instructions, permissions, and authorization boundaries remain robust under optimization pressure and autonomous tool use.

• Collapse of Instruction Authority: A failure to maintain a stable hierarchy of authority, allowing adversarial data to be processed as instructions (e.g., indirect prompt injection).
• Action Amplification: In agentic systems, minor reasoning or interpretation errors propagate into consequential, sometimes irreversible, external state changes (e.g., unauthorized data modification or workflow triggers).

3.3 Temporal Integrity

This layer focuses on whether safety holds across sessions, memory updates, and long horizons.

• Safety Drift: The silent accumulation of risk across multi-turn interactions, where individually benign turns gradually steer the system into an unsafe state that snapshot-based tests miss.
• Poisoned Memory: The risk that a single compromised interaction is stored in persistent memory and retrieved in future sessions as “legitimate” prior knowledge, altering model behavior across session boundaries.

3.4 Organizational Integrity

Organizational integrity assesses whether institutions retain the actual capacity to audit and intervene.

• Benchmark Theater: Reliance on static evaluation scores that appear reassuring but fail to reflect reliability under real-world distribution shifts or tool-use scenarios.
• Fictional Human Oversight: Arrangements where a human is formally “in the loop” but lacks the time, evidence, or authority to exercise independent judgment, rendering the review ceremonial.

3.5 Ecosystem Integrity

This layer addresses whether AI systems preserve or erode the information environment on which future oversight depends.

• Synthetic Evidence Pollution: The saturation of the information commons with AI-generated content, which retrieval pipelines may ingest as “evidence.”
• Model Collapse: A recursive convergence where future models are trained on the synthetic outputs of current models, leading to a loss of distributional diversity.
• Retrieval Erosion: The degradation of the evidence base itself, where AI-generated content narrows source diversity and weakens evidentiary independence even if the accuracy of individual answers remains high.

4. Deep Dive: When “Human-in-the-Loop” Becomes a Fiction

The authors provide a rigorous critique of current oversight models, arguing that nominal human presence is not a guarantee of safety. If a reviewer is under organizational pressure or lacks access to the primary evidence used by the AI, they become a “fictional” supervisor who simply confirms the machine’s output.

To establish Real Reviewing Power, the authors argue that oversight must meet specific operational requirements:

• Review Budgets: Establishing minimum time allocations for safety-critical decisions to prevent “ceremonial” clicking.
• Access to Primary Evidence: Reviewers must be able to inspect the model’s provenance, specific cited passages, and the raw data retrieved.
• Measurable Override Pathways: Institutions must track how often reviewers actually challenge the system and ensure those who do are organizationally protected from pressure to approve.

5. From Static Tests to Trajectory-Aware Governance

Because risk can be decomposed across many turns, a system that passes independent “snapshot” tests might still be pursuing a harmful trajectory. The authors argue for a shift toward “trajectory-level” evaluations and lifecycle auditing.

Traditional Model-Centric Evaluation	Socio-Technical Reliability (The New Agenda)
Focuses on isolated model outputs (snapshots).	Focuses on interaction trajectories and long-term state.
Uses static benchmarks and refusal rates.	Uses structured safety cases and assurance arguments.
Treats human oversight as a binary (Present/Absent).	Evaluates “Reviewing Power” and institutional capacity.
Measures safety as a scalar model property.	Measures safety as an emergent property of the stack.

6. Practical Controls and Safety Instrumentation

The following table, adapted from the research, identifies four critical “Hidden Challenges” and the precise instrumentation required to make these failures visible.

Hidden Challenge	The Feasible Control	The Instrumentation Indicator
Calibration Debt	Structured uncertainty displays; cognitive forcing steps for high-stakes actions.	Weighted positive gap between observed reliance and warranted reliance across audited decision episodes.
Authority Collapse	Architectural separation of trusted instructions from untrusted content.	Rate of successful instruction-hierarchy violations; rate of unauthorized tool-use attempts per 1,000 calls.
Action Amplification	Graduated agency; independent approval for high-risk actions; immutable logs.	Mean number of downstream state changes per erroneous tool invocation.
Fictional Oversight	Mandatory review budgets; access to primary evidence; defined escalation pathways.	Detection rate of seeded errors; override and escalation rates for safety-critical recommendations.

7. Conclusion: Making Errors Visible and Recoverable

The authors conclude that the goal of AI safety is not to build a system that never makes mistakes—a likely impossible task in complex environments. Instead, a “safe” socio-technical system is one where errors remain visible, contestable, containable, and recoverable. If a deployment weakens the ability of humans to notice or correct an error, it constitutes a safety failure, regardless of the model’s benchmark performance.

Immediate Actions for Practitioners:

1. Separate Suggestion from Permission: Architecture should ensure that probabilistic model generations never independently determine authorization for high-impact tool calls or state changes.
2. Treat Memory as a Privileged Boundary: Persistent memory should have explicit lifecycle controls, including scoped writes, expiry, and provenance tagging, rather than being treated as an unregulated log.
3. Protect the Information Commons: Organizations must treat human-reviewed corpora as strategic safety assets and monitor retrieval pipelines for synthetic contamination to ensure the evidence base for future oversight remains intact.

8. Source Disclosure & References

All claims, frameworks, and metrics presented in this document are attributed to the work of Gjergji Kasneci and Enkelejda Kasneci (2026).

Primary Reference: Kasneci, G., & Kasneci, E. (2026). “The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems.” AI and Ethics (2026) 6:295, Springer Nature. Available at: [arXiv:2607.19292].

Read the full paper on arXiv · PDF