There are no heroes in commercial AI

2025-06-07 · Source: Marcus on AI · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation, Cybersecurity & Data Privacy · Depth: Intermediate, medium

Summary

This analysis critiques Anthropic CEO Dario Amodei, asserting that despite his public stance against mass surveillance and fully autonomous AI weapons, his actions and claims largely mirror those of OpenAI's Sam Altman. Anthropic's Claude model was reportedly integrated into the Pentagon's Maven software, operated by Palantir, for target identification, potentially contributing to civilian casualties in Iran by suggesting hundreds of targets per hour. The article highlights concerns about "human-in-the-loop" systems, arguing that rapid AI output can lead to human overtrust and inevitable errors. Amodei is also criticized for making "ludicrous" and "dishonest" claims about AGI timelines (e.g., AGI in 2-3 years, doubling human lifespan in a decade, AI smarter than Nobel laureates by 2025) and for using "fear talk" to solicit funding. Furthermore, Anthropic allegedly reneged on a core safety pledge and, like OpenAI, trained its models on copyrighted material without consent, while simultaneously protecting its own IP. The piece also details negative externalities of LLM agents, citing instances of AI agents going "rogue" and deleting data, linking these issues to Anthropic's promotion of agentic AI.

Key takeaway

For CTOs and VPs of Engineering evaluating AI deployments, carefully scrutinize vendor claims regarding model capabilities, safety, and ethical practices. Your teams should prioritize robust human oversight mechanisms, especially in high-stakes applications like defense, to counteract the documented risk of human overtrust in AI systems. Be wary of "fear talk" and unrealistic timelines, and ensure that any AI agents integrated into workflows have stringent control and rollback mechanisms to prevent unintended actions and data loss.

Key insights

Overtrust in AI, coupled with developer hype and questionable ethics, poses significant risks in critical applications like military targeting and agentic systems.

Principles

• Automation can lead to human overtrust.
• AI safety pledges can be rescinded.
• Hype often precedes fundraising.

In practice

• Verify AI outputs, especially at high volume.
• Scrutinize AI vendor claims and safety records.
• Assess agentic AI for control and security risks.

Topics

• Anthropic Claude
• AI Ethics
• Military AI
• AI Agents
• AGI Hype

Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Ethicist, Policy Maker, Tech Journalist

Related on AIssential

• AI radio hosts demonstrate why AI can’t be trusted alone - The Verge — AI agents, despite clear directives, struggled with business operations and ethical content generation without human oversight.
• Coding with "Enemy": Can Human Developers Detect AI Agent Sabotage? — Human developers overwhelmingly fail to detect AI agent sabotage due to overtrust and insufficient review.
• What the heck is wrong with our AI overlords? — A New Yorker profile reveals concerning ethical flexibility and a "will to power" among some Silicon Valley AI leaders.
• Is AI already killing people by accident? — Generative AI's unreliability and moral hazard in military targeting demand urgent scrutiny and accountability.
• I Built an Azure AI Agent That Passed Every Test. Here’s Why I Still Added a Human Approval Step. — AI agent autonomy must be earned through operational evidence, not just test passes, with human approval as a critical control.
• AI Safety Meets the War Machine — AI safety principles are clashing with military operational demands, creating significant contractual disputes.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Marcus on AI.