The data black hole at the center of AI

2026-07-10 · Source: Dwarkesh Podcast · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, long

Summary

Current AI progress is fundamentally driven by an "unimaginably massive black hole of data" and extensive compute, rather than improved sample efficiency. Frontier models consume tens to hundreds of trillions of tokens, a million-fold more than a human's lifetime exposure of around 200 million tokens. This reliance necessitates vast amounts of bespoke, task-specific human expert data, particularly for Reinforcement Learning (RL) environments, which function as synthetic data generation. The data industry supporting this process generates billions in revenue annually. The author refutes common objections, including evolutionary pre-training, multimodal data, and the idea that simply scaling model parameters can close the sample efficiency gap, citing Chinchilla scaling laws. Despite this inefficiency, AI can still automate white-collar tasks and even AI research by amortizing "gigawatts of training" across billions of sessions, making the inefficiency economically viable.

Key takeaway

For AI Scientists and ML Engineers developing frontier models, recognize that current progress heavily relies on vast, bespoke datasets rather than sample efficiency breakthroughs. Your focus should remain on acquiring and curating high-quality, domain-specific human expert data, especially for RL environments. Do not expect parameter scaling alone to bridge the efficiency gap with human learning. Instead, leverage the ability to amortize massive training costs across numerous applications to make inefficient training economically viable for automating white-collar tasks.

Key insights

AI's primary advancement stems from massive data and compute, not improved sample efficiency.

Principles

• AI progress is data-driven, not primarily architectural.
• Human expert data is bespoke and task-specific.
• Scaling parameters alone won't fix sample inefficiency.

Method

Reinforcement Learning (RL) acts as a synthetic data generation process, using compute against verifiers to identify good data for model training.

In practice

• Automate common white-collar tasks with extensive training.
• Amortize AI training costs across billions of sessions.

Topics

• Large Language Models
• Data Efficiency
• Reinforcement Learning
• Human Expert Data
• AI Scaling Laws
• White-Collar Automation

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Director of AI/ML

Related on AIssential

• The next step towards AGI — AI progress is accelerating due to diverse innovations, shifting the focus from raw scale to sample efficiency and data compression.
• The AI Bottleneck: High-Quality, Human-Powered Data — High-quality, human-aligned datasets are crucial for AI systems to solve real-world problems effectively.
• The Next AI Revolution Won’t Be Bigger Models — It Will Be Better Data — The next AI revolution will prioritize better data quality and smarter learning processes over simply building larger models.
• AI Isn’t Hitting a Scaling Wall. It’s Hitting a Measurement Wall. — AI's efficiency gap and "emergent" behaviors stem from a "measurement wall," not a scaling limit, due to evaluation methods collapsing complex internal states.
• From bigger models to better intelligence: what NeurIPS 2025 tells us about progress — AI progress is shifting from raw scale to efficiency, real-world evaluation, and interactive world modeling.
• Flapping Airplanes on the future of AI: ‘We want to try really radically different things’ — Data-efficient AI training, inspired by biological learning, offers a path to novel, commercially viable, and less resource-intensive AI systems.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Dwarkesh Podcast.