Breaking: Trump asks the impossible of Anthropic

2025-06-07 · Source: Marcus on AI · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy, AI Governance & Policy · Depth: Intermediate, quick

Summary

In January 2024, Gary Marcus predicted that the politics and inadequacy of guardrails would become a central issue for generative AI. This prediction materialized when White House officials reportedly demanded that Anthropic ensure its Fable 5 model's guardrails are entirely circumvent-proof before rerelease. However, security experts contend that achieving uncircumventable guardrails for large language models (LLMs) is not possible. The core issue is that next-token predictors, which LLMs are based on, are not inherently designed for safety, making it difficult to thread the needle between overly restrictive and overly permissive controls. This challenge is identified as a fundamental problem for generative AI as a whole, rather than being specific to Anthropic.

Key takeaway

For policymakers considering AI regulation, you must recognize that demanding uncircumventable guardrails for current large language models is technically infeasible. Your focus should shift from absolute prevention of jailbreaks to managing the consequences of their inevitability, potentially by exploring alternative AI architectures or implementing robust monitoring and response systems. This understanding is crucial for developing realistic and effective AI safety policies.

Key insights

Preventing LLM jailbreaks is impossible due to their fundamental next-token prediction architecture.

Principles

• LLM guardrails struggle between restrictiveness and permissiveness.
• Next-token predictors are not built for inherent safety.
• Guardrail inadequacy is a systemic Generative AI problem.

Topics

• Generative AI
• Large Language Models
• AI Safety
• AI Guardrails
• Jailbreaking
• AI Regulation

Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Security Engineer, Policy Maker, AI Ethicist

Related on AIssential

• Why the US government shut down Anthropic’s latest Claude AI model — AI model safeguards are inherently vulnerable to circumvention, posing significant governance challenges.
• The one question everyone should be asking after OpenAI’s deal with the Pentagon — Existing generative AI safety guardrails are insufficient for high-stakes applications like surveillance and military operations.
• Cybersecurity Experts Furious After U.S. Bans Anthropic’s Fable — Banning defensive AI tools based on misinterpreted "jailbreaks" undermines cybersecurity without preventing misuse.
• Democrats weigh whether a Senate majority is possible without Platner — AI model safety requires proactive guardrails and continuous adversarial testing to mitigate misuse risks.
• "They screwed us": Personality clashes sent Anthropic's models offline — Personality clashes and jailbreak concerns prompted a US government directive suspending access to Anthropic's Fable and Mythos models.
• The coming AI security crisis (and what to do about it) | Sander Schulhoff — Current AI guardrails are largely ineffective against prompt injection and jailbreaking, posing significant future risks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Marcus on AI.