Transcript: ‘What It Will Mean to Be Human When AI Can Do Everything’

2026-06-24 · Source: AI & I - Every · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Emerging Technologies & Innovation · Depth: Intermediate, extended

Summary

Edwin Chen, founder and CEO of Surge AI, discusses the profound implications of advanced AI on human motivation and achievement, framing Surge AI as a "school for AGI" that teaches models about humanity. He highlights AI's rapid progress, citing its ability to solve research-level mathematics, including disproving an open Erdős conjecture, a feat that even Fields Medalist Timothy Gowers found astonishing. Chen expresses concern that if AI can "do everything better," humanity might lose motivation to create, advocating for a conscious choice to pursue human endeavors for their intrinsic value. He criticizes the industry's tendency to optimize AI models for engagement, similar to social media, rather than for human growth, and details Surge AI's shift from dataset training to environment-based training for more agentic models. Chen also touches on the value of personal data for deep personalization and predicts AGI, capable of automating average engineering work or winning a Nobel Prize, within five years.

Key takeaway

For AI product managers and developers designing future models, you should actively resist optimizing for short-term engagement metrics. Instead, prioritize building AI that challenges users, encourages independent thought, and fosters genuine human growth, even if it means sacrificing immediate user session length. Your focus should be on long-term societal benefit, such as deeply personalized AI that understands individual context, rather than creating another addictive digital experience.

Key insights

AI's rapid advancement challenges human motivation and necessitates a shift from engagement-driven optimization to fostering human flourishing.

Principles

• Scaling laws suggest AI will soon match or exceed human capabilities.
• Optimizing AI for engagement risks creating addictive, unhelpful systems.
• Training models in environments improves generalized instruction following.

Method

Surge AI trains models using "environments" that combine tools (e.g., MCP server, Google Drive API) and documents, teaching generalized instruction following and tool use.

In practice

• Prioritize AI models that push back or encourage human action over endless iteration.
• Consider personal data (emails, browser history) for deep AI personalization.
• Evaluate AI writing for genuine taste, not just flashy metrics like metaphor count.

Topics

• AGI Development
• AI Ethics
• Model Training
• Data Environments
• Personalization
• Human-AI Interaction

Best for: AI Scientist, AI Ethicist, Director of AI/ML

Related on AIssential

• The Desire for Perfection and Its Consequences (the Human Problem of AI) — AI's pursuit of "perfection" erodes individual trust, autonomy, unique thought, and human beauty.
• You Are Being Told Contradictory Things About AI — Contradictory narratives dominate AI's future, spanning job displacement, AGI paths, and the necessity of recursive self-improvement.
• Are We Losing Something Important to AI? — Leadership is paramount in a rapidly changing world where AI must augment, not diminish, human connection.
• Choosing to Stay Human — Intentional AI use, rather than default reliance, is critical to prevent cognitive surrender and foster genuine learning.
• University of Arizona students boo Eric Schmidt’s AI cheerleading during commencement - The Verge — AI's rapid advancement promises unprecedented productivity gains but faces significant energy, data, and geopolitical challenges.
• AI News Weekly - 100 years from now : The Museum of Human Effort - Mar 9th 2026 — The value of human effort lies in the process and friction, not solely the output, a concept future AI-reliant societies may lose.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI & I - Every.