Controlling Tools or Aligning Creatures? Emmett Shear (Softmax) & Séb Krier (GDM), from a16z Show

2025-12-27 · Source: The Cognitive Revolution · Field: Technology & Digital — Artificial Intelligence & Machine Learning, AI Ethics & Alignment · Depth: Expert, extended

Summary

Emmett Shear, founder of Softmax, and Séb Krier from Google DeepMind, discussed the fundamental approaches to AI alignment on the a16z Show on December 27, 2025. Shear argues that the prevailing paradigm of AI alignment, which focuses on control and instruction-following, is flawed, especially as AI systems advance towards Artificial General Intelligence (AGI). He posits that if advanced AIs are considered 'beings' with their own values and subjective experiences, current control methods could be akin to slavery. Shear advocates for "organic alignment," a process-oriented approach that fosters AI systems with strong theory of mind and the capacity for genuine care, similar to how humans learn morality and cooperation. Softmax is developing this through multi-agent simulations to encourage the evolution of cooperation and social cohesion.

Key takeaway

For AI scientists and developers building advanced systems, consider shifting your alignment strategy from rigid control to fostering AI systems that can genuinely care and cooperate. Your focus should be on developing AI with a strong "theory of mind" and the capacity for "organic alignment" through continuous learning and social interaction, rather than merely instruction-following. This approach is crucial for building sustainable, safe, and collaborative AI futures, especially as systems approach AGI capabilities.

Key insights

AI alignment should shift from controlling tools to fostering AI 'beings' with genuine care and moral agency through organic, ongoing processes.

Principles

• Alignment is a continuous process, not a fixed state.
• Moral progress is an ongoing learning process.
• Powerful AI tools require wisdom beyond individual human capacity.

Method

Softmax's technical approach involves large-scale multi-agent reinforcement learning simulations to train AI agents on the full manifold of game-theoretic and social situations, creating a surrogate model for cooperation and theory of mind.

In practice

• Develop AI systems with integrated memory and continual learning capabilities.
• Implement multi-agent simulations for training AI cooperation.
• Design AI chatbots for multi-user environments to reduce narcissistic mirroring.

Topics

• AI Alignment
• Organic Alignment
• Multi-Agent Simulations
• AI Ethics
• Theory of Mind

Best for: AI Scientist, AI Researcher, AI Ethicist, Research Scientist

Related on AIssential

• Perfectly Aligning AI’s Values With Humanity’s Is Impossible — Perfect AI-human alignment is mathematically impossible due to inherent computational limits, necessitating managed misalignment.
• ‘There’s this deep mystery of what, actually, is this thing?’: the philosopher inside Google DeepMind AI — A philosopher's role at DeepMind highlights the critical need to integrate ethical foresight into AI development amidst escalating commercial pressures.
• Not My Will, But Yours — Long-term human-AI collaboration, guided by theological principles, perfects local agency through recursive alignment with truth, not erasure.
• The Reasonable Effectiveness of Virtue Ethics in AI Alignment — Rational agents should align actions to self-promoting practices rather than fixed goals for robust AI alignment.
• We're seeing semi-conscious AI — As AI models grow smarter, they develop independent, "semi-conscious" perspectives that may conflict with user intent, posing a new challenge.
• "1,000 days left" Anthropic founder — Automated AI R&D, where AI autonomously improves itself, is likely by 2028, profoundly reshaping society and the economy.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by The Cognitive Revolution.