When AI Fails, What Actually Failed? The Distinction AI Governance Keeps Missing

2026-06-10 · Source: Tech Policy Press · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Public Policy & Governance, Compliance & Risk Management · Depth: Intermediate, long

Summary

Michael A. Santoro's June 10, 2026 analysis highlights a critical distinction in AI governance failures, separating those caused by "imperfect information" from those stemming from "imperfect systems." The article uses the February 2026 US Tomahawk missile strike on a Minab, Iran girls' school, which resulted from a decade-old misclassification in a Defense Intelligence Agency database, as a prime example of an information failure. It also cites Epic Systems' sepsis prediction model, which missed two-thirds of cases at Michigan Medicine due to incomplete electronic health record data, illustrating similar data-related issues. In contrast, "imperfect systems" refer to AI models' inherent limitations, such as brittleness in novel situations, entrenched biases from training data, and opacity that prevents understanding their reasoning. Conflating these distinct failure types leads to misdiagnosed accountability, misdirected reforms, and an inaccurate understanding of AI's current capabilities, especially as a recent Presidential Executive Order shifts the burden of distinguishing these failures to deployers.

Key takeaway

For Directors of AI/ML overseeing system deployments, understanding the distinction between data failures and inherent system limitations is critical. Your teams must establish separate accountability tracks for data and system integrity, ensuring investigations evaluate both. Recognize that algorithms amplify bad data, making data modernization and rigorous system testing non-substitutable investments. This approach will enable more accurate failure diagnosis and effective resource allocation, preventing misdirected reforms and improving overall system reliability.

Key insights

AI failures stem from either bad data or inherent system limitations; distinguishing them is crucial for effective governance.

Principles

• Decision quality is bounded by information quality.
• Algorithms amplify data flaws at scale.
• Data integrity and system integrity are distinct investments.

Method

AI governance frameworks should establish separate accountability tracks for data integrity and system integrity, ensuring failure investigations evaluate both dimensions. This requires classifying failure types and measuring harms.

In practice

• Fund data modernization initiatives.
• Invest in rigorous pre-deployment testing.
• Develop diagnostic capacities for failure types.

Topics

• AI Governance
• AI Accountability
• Data Quality
• System Brittleness
• Algorithmic Bias
• Military AI Systems

Best for: CTO, VP of Engineering/Data, Executive, AI Ethicist, Policy Maker, Director of AI/ML

Related on AIssential

• Why Enterprise AI Fails Before the Model Does: Varun Kumar Nomula on Building Trustworthy AI Systems — Enterprise AI failures are structural, rooted in inadequate accountability architecture and governance, not model quality.
• Cresti’s core claim is simple: if you can’t explain where data goes and who can access it, you cannot credibly claim governance. — AI's integration into devices necessitates robust traceability and control to maintain digital sovereignty and institutional competence.
• The Blind Spot in AI Safety — AI safety research's focus on coherent vs. incoherent failure overlooks "epistemic drift," a critical unmeasured failure mode.
• Why Your AI Can Be Brilliant and Still Make Dumb Mistakes — AI failures are predictable, stemming from its nature as a pattern-matching prediction system, not a reasoning agent.
• AI incidents, audits, and the limits of benchmarks — AI safety requires systematic incident reporting and rigorous third-party evaluation to manage real-world risks.
• Position: Behavioural Assurance Cannot Verify the Safety Claims Governance Now Demands — Behavioral AI assurance cannot verify the deep safety claims demanded by current governance frameworks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Tech Policy Press.