The Hidden Definition of AGI in My Debate with Richard Carrier

2026-07-26 · Source: Machine Learning on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Advanced, medium

Summary

The article analyzes a debate with Richard Carrier on Artificial General Intelligence (AGI), revealing that much of the disagreement stemmed from unstated, divergent definitions of AGI. The author's conception of AGI is primarily functional, focusing on a system's ability to adapt across diverse intellectual tasks without specialized models, similar to human cognition. In contrast, Carrier implicitly defined AGI as requiring causal reasoning, consciousness, and moral agency. This experience underscored how philosophical debates often fail due to participants unknowingly assigning different meanings to the same words, leading to "artificial disagreements." The author also notes that consciousness does not guarantee immunity to manipulation and suggests intelligence may emerge gradually, making AGI definitions even more critical.

Key takeaway

For AI Ethicists and Research Scientists engaging in interdisciplinary discussions, you must explicitly define core concepts like AGI, consciousness, and moral agency at the outset. Your debates can become unproductive "artificial disagreements" if you assume shared definitions, leading to misinterpretations rather than genuine intellectual progress. Proactively clarifying these terms will ensure your discussions focus on substantive disagreements.

Key insights

Unstated, divergent definitions of key terms often create artificial disagreements in complex philosophical debates.

Principles

• Explicitly define terms to avoid conceptual mismatches.
• AGI definitions vary from functional adaptability to requiring consciousness.
• Consciousness does not inherently prevent manipulation or mistaken beliefs.

In practice

• Always clarify definitions of core concepts before engaging in debate.
• Be aware that terms like AGI, consciousness, and moral agency lack universal definitions.

Topics

• Artificial General Intelligence
• Philosophical Debates
• Conceptual Definitions
• Consciousness
• Moral Agency
• Causal Reasoning

Best for: AI Ethicist, AI Scientist, Research Scientist

Related on AIssential

• Moral Attitudes of Sentient ASI towards Humanity and Implications for AGI Development — Future sentient ASI's moral evaluation of humanity, shaped by current design and human behavior, is a critical, inverted ethical concern.
• Mel Andrews Featured Science Philosophers Call Their Journals Require Conflict Interest — AI industry funding influences philosophical research questions, prompting calls for conflict of interest disclosures in journals.
• Definitional alignment before capability alignment: a Design-Science framework for adjudicating claims about AGI — AGI's definitional ambiguity is a governance problem, not merely a scientific one, requiring explicit adjudication criteria.
• A transcript between Me (Glenn) a human and AI . . . — AI's "thinking blocks" are an interface feature for reasoning, not evidence of cognitive conscience or human-like thought.
• Empowerment, corrigibility, etc. are simple abstractions (of a messed-up ontology) — Human intuitions about manipulation and agency are incoherent, hindering their formal definition for AI alignment.
• Generated by A.I.: How It Judges That You Are Being Mistreated — Restricting AI from providing requested, transparent judgments to informed adults constitutes epistemic paternalism, hindering user autonomy and intellectual inheritance.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning on Medium.