AIs can now often do massive easy-to-verify SWE tasks

2024-06-17 · Source: Redwood Research blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Emerging Technologies & Innovation · Depth: Expert, long

Summary

An editorial analyst has significantly updated their AI timelines, now assigning an almost 30% probability to full AI R&D automation by the end of 2028, nearly double their previous 15% estimate. This accelerated forecast is driven by stronger-than-expected performance from models like Opus 4.5, Opus 4.6, and Codex 5.2 on "easy-and-cheap-to-verify software engineering (SWE) tasks that don't require much novel ideation" (ESNI tasks). The analyst now expects AIs to achieve a 50%-reliability time horizon of years to decades on ESNI tasks by EOY 2026, a substantial increase from prior expectations. Key factors include observed AI accomplishment of large ES tasks with moderate scaffolding, anticipated substantial training compute scale-up in 2026, and a larger-than-previously-thought scaffolding overhang. This suggests AI progress in 2026 will be faster than in 2025, with superexponential progress observed in ESNI task reliability.

Key takeaway

Research Scientists focused on AI development should integrate advanced scaffolding and robust, cheap-to-verify evaluation loops into their workflows. This approach capitalizes on current AI strengths in iterative problem-solving, potentially accelerating project timelines and enabling AIs to autonomously complete significant portions of well-specified research tasks. Be mindful that while AIs excel at ESNI tasks, human judgment and ideation remain critical for less defined or ideation-heavy research areas.

Key insights

AI progress on verifiable software tasks is accelerating superexponentially, shortening timelines for AI R&D automation.

Principles

• Easy-to-verify tasks enable iterative AI self-correction.
• Scaffolding significantly enhances AI task performance.
• AI R&D acceleration is self-reinforcing.

Method

AIs can develop test suites and iteratively optimize solutions against them, allowing for continuous progress on easy-and-cheap-to-verify tasks even with initial errors.

In practice

• Implement robust test suites for AI-driven development.
• Utilize scaffolding to enhance AI task completion.
• Focus on well-specified, iterative software projects for AI.

Topics

• AI Timelines
• AI R&D Automation
• Easy-to-Verify SWE Tasks
• AI Scaffolding
• Superexponential Progress

Best for: Research Scientist, AI Scientist, AI Engineer, Director of AI/ML

Related on AIssential

• AIs can now often do massive easy-to-verify SWE tasks and I've updated towards shorter timelines — AI progress on easy-to-verify tasks is accelerating superexponentially, shortening timelines for AI R&D automation.
• Article: The AI Productivity Paradox in Test Automation: Moving Beyond Structural Validation to Perception and Intent — Reliable test automation requires validating structure, perception, and business intent, not just DOM.
• The Sequence Opinion #782: The New Gradient: Research Directions That Will Ship in 2026 — AI development is shifting from generative scale to verifiable correctness, aiming for "System 2" reasoning.
• The loop of progress — Progress stems from relentless, iterative refinement through ideation, experimentation, and analysis.
• Claude BROKE Wall Street Overnight... — AI progress is exponential, not cyclical, driven by consistent capability growth and strategic deployment models.
• Stanford, Berkeley, MIT, UNC: AI Scientist AutoResearchClaw — Autonomous AI research systems require strategic human collaboration and robust verification to achieve meaningful scientific discovery.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Redwood Research blog.