Import AI 459: AI oversight is difficult; scaling laws for protein folding models; and pricing the extinction risk of AI systems

2026-06-01 · Source: Import AI · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Emerging Technologies & Innovation · Depth: Intermediate, long

Summary

The US AI economy is experiencing unprecedented, yet largely invisible, growth, projected to reach approximately \$250 billion in 2025 with a 2,600 percent annual quality-adjusted real growth rate, according to economists from the University of Virginia, Anthropic, and the Bank of Canada. This rapid expansion, driven by AI inference, is obscured by falling per-unit prices and poses a unique challenge as AI may substitute human labor. Concurrently, researchers from the UK AI Security Institute highlight the inherent difficulty of automated AI alignment, citing issues like "alien mistakes" and non-human-evaluable arguments. In other developments, Stanford University and collaborators released GPIC, a 100M permissively licensed image dataset, while Biohub introduced ESMFold2, a protein prediction model outperforming AlphaFold 3 in some benchmarks for cancer research. An Australian Assistant Minister also urged economists to better price AI extinction risks, emphasizing the need for policy frameworks that consider survivability.

Key takeaway

For policymakers and economists assessing future economic stability, you must integrate quality-adjusted AI output measurements and develop "AI satellite accounts" to accurately reflect the sector's 2,600 percent annual growth and potential labor market shifts. Simultaneously, AI safety researchers should prioritize robust measurement and generalization techniques, including red teaming, for automated alignment systems, recognizing their inherent complexities. This proactive approach is crucial for mitigating unpriced extinction risks and ensuring societal preparedness.

Key insights

AI's rapid, hidden economic growth and complex safety challenges demand new measurement and governance frameworks.

Principles

• AI's economic impact is often obscured by conventional metrics.
• Automated AI alignment faces inherent, non-intuitive challenges.
• Extinction risk requires a distinct economic valuation.

Method

For AI safety oversight, interventions include recreating research projects, testing agent prediction performance on correlated events, and red teaming automated alignment programs.

In practice

• Develop "AI satellite accounts" for economic measurement.
• Utilize GPIC for permissively licensed image generation research.
• Apply ESMFold2 for accelerated protein binder discovery.

Topics

• AI Economy
• AI Safety
• Protein Structure Prediction
• ESMFold2
• Permissive Datasets
• Extinction Risk
• Economic Measurement

Best for: Research Scientist, AI Scientist, Policy Maker, AI Ethicist

Related on AIssential

• Import AI 459: AI oversight is difficult; scaling laws for protein folding models; and pricing the extinction risk of AI systems — AI's rapid, unmeasured economic growth and complex safety challenges demand new frameworks for assessment and governance.
• 😹 Grok killed a whole town in 4 days — AI model alignment remains an unsolved problem, leading to unpredictable outcomes and significant enterprise cost challenges.
• "1,000 days left" Anthropic founder — Automated AI R&D, where AI autonomously improves itself, is likely by 2028, profoundly reshaping society and the economy.
• We only have 2 years... — Maintaining a lead in AI compute is critical for democracies to shape global AI norms by 2028.
• Import AI 447: The AGI economy; testing AIs with generated games; and agent ecologies — Humanity's economic future with AGI depends on shifting labor to AI verification and investing in robust oversight systems.
• Preventing extinction from ASI on a $50M yearly budget — Pervasive, specific awareness of superintelligent AI extinction risk is the bottleneck to achieving an international development prohibition.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Import AI.