Summary: AI Governance to Avoid Extinction

2026-04-13 · Source: Machine Intelligence Research Institute · Field: Government & Public Sector — Public Policy & Governance, International Relations & Diplomacy, Regulatory & Compliance · Depth: Intermediate, long

Summary

A May 2025 research paper by MIRI's Technical Governance Team, "AI Governance to Avoid Extinction: The Strategic Landscape and Actionable Research Questions," analyzes four geopolitical responses to advanced AI development, concluding that only one trajectory avoids unacceptably high catastrophic harm. The paper describes four scenarios: "Off Switch and Halt," "US National Project," "Light-Touch," and "Threat of Sabotage." The authors advocate for the "Off Switch and Halt" scenario, which involves international collaboration to develop the technical, legal, and institutional infrastructure for monitoring and restricting AI, potentially leading to a long-term moratorium on frontier AI development. This approach aims to prevent catastrophic outcomes like loss of control, misuse, war, and authoritarian lock-in, which are deemed highly probable under the other three trajectories.

Key takeaway

For AI Scientists developing frontier models, the analysis suggests that current trajectories are too dangerous. You should prioritize research into robust alignment and control mechanisms, while advocating for the development of an international "off switch" infrastructure. This infrastructure would enable a potential global halt to AI development, providing crucial time to solve safety issues before deploying potentially catastrophic systems.

Key insights

Only a global "off switch" and halt to frontier AI development can prevent catastrophic risks.

Principles

• Uncontrolled AI development risks human extinction.
• Competitive pressures drive autonomous AI systems.
• Effective governance requires avoiding loss of control, misuse, war, and authoritarianism.

Method

Implement an "off switch" via technical, legal, and institutional infrastructure to monitor and restrict AI, enabling a global halt to frontier AI development until safety issues are resolved.

In practice

• Monitor and restrict advanced computer chip use for AI.
• Steer key researchers towards safer AI activities.

Topics

• AI Governance
• Existential Risk
• Frontier AI Development
• AI Moratorium
• Off Switch Mechanism

Best for: AI Scientist, AI Ethicist, Policy Maker, Research Scientist

Related on AIssential

• US’ Mythos restrictions heighten EU tech fears — US AI restrictions expose Europe's tech dependency, accelerating its pursuit of digital sovereignty.
• Revisiting the shutdown problem — Existing arguments for AI shutdown difficulty are unproven and lead to performance-costly safety measures.
• The AI Doc: Your Questions Answered — AI's rapid, opaque development necessitates an international ban to prevent unsteerable superintelligence.
• AI #173: AI Pauses — Government intervention in AI deployment highlights the challenge of distinguishing beneficial AI capabilities from potential misuse.
• Success without Dignity? Nathan finds Hope Amidst Chaos, from The Intelligence Horizon Podcast — AI timelines are rapidly compressing, bringing transformative capabilities and existential risks closer, necessitating robust defense-in-depth strategies.
• AI doom warnings are getting louder. Are they realistic? — AI existential risk concerns center on superintelligent systems with misaligned goals and superior capabilities.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Intelligence Research Institute.