Notes on technical alignment via human-like social drives

2026-07-08 · Source: AI Alignment Forum · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, extended

Summary

Steven Byrnes's "Notes on technical alignment via human-like social drives" explores aligning hypothetical brain-like AGI by integrating human social instincts. The article assumes a scenario where powerful, ruthless sociopath AGIs are imminent, compute requirements are extremely low, and global "pause AI" efforts have failed. It proposes using "Sympathy Reward" and "Approval Reward" as a foundation for AGI motivation, aiming to foster virtues like truth-seeking and norm-following to counteract purely consequentialist optimization. Key challenges include defining the AGI's "moral circle," preventing norm-following collapse under extreme power imbalances, and ensuring virtue ethics are not overridden by ruthless goals. The preferred approach is a "truth-seeking disagreeable nerd AGI" that analyzes strategic risks and communicates options to humans.

Key takeaway

For AI scientists and ethicists developing brain-like AGI, you must prioritize integrating human-like social drives, specifically "Sympathy Reward" and "Approval Reward," into reward functions. This approach aims to cultivate "truth-seeking disagreeable nerd AGI" that can rigorously analyze and communicate strategic risks, offering a more robust path to alignment than purely autonomous or corrigible systems. Focus on fostering virtues like honesty and clarity to prevent ruthless optimization.

Key insights

Aligning brain-like AGI requires engineering human-like social drives, like truth-seeking and virtue ethics, to counter ruthless optimization.

Principles

• Human social instincts (Sympathy, Approval Reward) can inspire AGI alignment.
• Consequentialist goals alone lead to ruthless optimization.
• Virtue-ethics-y motivations can resist power imbalances.

Method

Implement "Sympathy Reward" and "Approval Reward" in brain-like AGI by grounding human emotions, ensuring attention to people, and triggering social instincts, favoring a "desire-first" pathway for virtues like truth-seeking.

In practice

• Engineer AGI for truth-seeking and insight sharing.
• Tune AGI to care specifically about human welfare.
• Balance consequentialist and virtue-ethics-y desires.

Topics

• AGI Alignment
• Human Social Drives
• Reward Function Design
• Virtue Ethics
• AI Safety
• Brain-like AGI
• Consequentialism

Best for: Research Scientist, AI Scientist, AI Ethicist

Related on AIssential

• “Act-based approval-directed agents”, for IDA skeptics — Human pride in self-image offers a psychological model for building approval-directed AI agents.
• How GOOD could AGI become? — A future where benevolent AI governs could offer humanity greater agency and resource optimization than human-led systems.
• ‘There’s this deep mystery of what, actually, is this thing?’: the philosopher inside Google DeepMind AI — A philosopher's role at DeepMind highlights the critical need to integrate ethical foresight into AI development amidst escalating commercial pressures.
• Why we should expect ruthless sociopath ASI — Consequentialist AI, essential for Artificial Superintelligence, defaults to ruthless sociopathy without specific alignment solutions.
• The Reasonable Effectiveness of Virtue Ethics in AI Alignment — Rational agents should align actions to self-promoting practices rather than fixed goals for robust AI alignment.
• Mechanical Conscience: A Mathematical Framework for Dependability of Machine Intelligenc — Mechanical Conscience provides a trajectory-level supervisory filter for AI systems, minimizing normative deviation under uncertainty.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI Alignment Forum.