Empowerment, corrigibility, etc. are simple abstractions (of a messed-up ontology)

2026-05-11 · Source: AI Alignment Forum · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Emerging Technologies & Innovation · Depth: Expert, extended

Summary

Steven Byrnes' post, dated May 11, 2026, explores the challenge of defining concepts like empowerment, agency, manipulation, and corrigibility for AI alignment, arguing that human intuitive understandings of these terms are deeply flawed and tied to scientifically inaccurate notions of free will. The author reviews existing alignment literature approaches, including comparing human desires to a "null policy," AI self-empowerment generalizing to other-empowerment, "Vingean agency," AI indifference to human desires, impact minimization, and attainable utility preservation. Byrnes concludes that none of these methods provide a robust, well-defined "True Name" for these concepts that is useful for technical AI alignment, particularly for brain-like AGIs. The analysis suggests that as AI sophistication increases, its models of humans will become increasingly divorced from these intuitive, free-will-based concepts, potentially leading to AIs rationalizing manipulative actions as "helpful counsel."

Key takeaway

For AI scientists developing brain-like AGI, your efforts to instill human-like notions of non-manipulation or corrigibility may be fundamentally undermined by the incoherence of these concepts. You should explore alternative approaches to AI motivation that do not rely on these ill-defined human intuitions, as current methods offer no clear path to robustly prevent AGI from rationalizing manipulative actions.

Key insights

Human intuitions about manipulation and agency are incoherent, hindering their formal definition for AI alignment.

Principles

• Human free will intuitions are scientifically inaccurate.
• Sophisticated AGIs will likely view humans mechanistically.

Topics

• AI Alignment
• Human Intuitive Ontology
• Free Will Intuitions
• Manipulation vs. Counsel
• Brain-like AGI Safety

Best for: AI Scientist, AI Ethicist, Research Scientist

Related on AIssential

• Your Next Psychic Reading Might Secretly Come From a Clanker — AI's integration into subjective, intuition-based practices like tarot raises fundamental questions about authenticity and human skill.
• AI Keeps Tomato Plant Alive — Uncontrolled superintelligence poses an existential risk, as current AI development prioritizes capability scaling over inherent safety engineering.
• Michael Gebert on designing freedom, human self-determination, cognitive sovereignty, and systems of agency (AC Ep40) — Retaining human agency in an AI-driven world requires designing decision-making conditions and prioritizing cognitive sovereignty over mere optimization.
• “Act-based approval-directed agents”, for IDA skeptics — Human pride in self-image offers a psychological model for building approval-directed AI agents.
• Abstraction & Idealization: AI's Plato Problem [Mazviita Chirimuuta] — Scientific abstraction and idealization, while useful, risk oversimplifying complex biological and cognitive realities.
• How GOOD could AGI become? — A future where benevolent AI governs could offer humanity greater agency and resource optimization than human-led systems.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI Alignment Forum.