Is AI already killing people by accident?

2025-06-07 · Source: Marcus on AI · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Emerging Technologies & Innovation · Depth: Intermediate, short

Summary

A recent incident involving a mistargeting that killed nearly 150 school children in Iran has prompted questions about the potential role of AI in military operations. While direct evidence linking AI to this specific event is unavailable and likely to remain so, the broader discussion highlights significant concerns regarding the deployment of generative AI in military contexts. Current generative AI models exhibit persistent problems with reasoning, visual cognition, and common sense, as demonstrated by studies from Anh Totti Nguyen and various online examples. The article emphasizes that without empirical studies on collateral damage, the true impact of AI in military targeting remains unknown. Beyond technical limitations, a critical moral problem arises: the potential for militaries to use AI to obscure accountability for civilian casualties, shifting blame from human decision-makers to algorithms.

Key takeaway

For AI Ethicists and Policy Makers evaluating military AI deployment, you must prioritize rigorous empirical studies on AI's real-world impact on collateral damage. Insist on clear human accountability frameworks, ensuring that decisions regarding acceptable casualty rates and algorithm limitations remain with human operators, not delegated to unreliable AI systems. The current rush to integrate AI into military applications is premature given its fundamental lack of reliability.

Key insights

Generative AI's unreliability and moral hazard in military targeting demand urgent scrutiny and accountability.

Principles

• AI unreliability causes real-world harm.
• Human users bear moral responsibility for AI outcomes.

In practice

• Conduct empirical studies on AI's collateral damage.
• Define acceptable casualty rates before AI deployment.

Topics

• Military AI
• AI Ethics
• Generative AI Reliability
• Targeting Systems
• AI Accountability

Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Ethicist, Policy Maker, Research Scientist

Related on AIssential

• AI Warfare Is Outpacing Our Ability to Control It — AI in warfare outpaces governance, leading to civilian harm, automation bias, and ethical dilemmas.
• Code Red for Humanity? — Unreliable Generative AI, if deployed in military decision-making, poses an immediate and catastrophic global risk.
• Deadline Day for Autonomous AI Weapons & Mass Surveillance — AI reliability and ethical governance are critical for deploying autonomous systems, especially in military and surveillance contexts.
• Don’t blame AI for the Iran school bombing | Letters — Human responsibility for AI actions must be maintained by avoiding anthropomorphic language and systemic blame.
• An Open Letter to Anthropic — Giving AI a visual identity can introduce psychological friction, making it harder to use AI in military kill chains.
• As AI accuracy increases, humans may become increasingly dependent on algorithmic “proactive conclusions,” potentially leading to the atrophy of “trained judgment”,... — Proactive AI systems can influence human decisions, necessitating ethical governance to preserve agency and mitigate risks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Marcus on AI.