AI's capability improvements haven't come from it getting less affordable

2024-06-17 · Source: Redwood Research blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, long

Summary

An analysis of METR's time horizon data indicates that the rising inference costs of frontier AI models do not signify increasing expense relative to human labor, but rather reflect models completing longer tasks. The study defines "cost ratio" as AI inference cost divided by human cost for the same task, finding that this ratio at a model's 50% reliability time horizon has not increased across successive frontier models. Furthermore, longer tasks do not exhibit higher cost ratios than shorter ones among successfully completed tasks. The analysis also demonstrates that capping AI spending per task at a fraction of human cost (e.g., 1/32x) barely slows time horizon trends, which continue to double approximately every three months. This contradicts a previous conclusion by Toby Ord, whose methodology is argued to significantly overestimate hourly model costs by conflating per-task budget with the 50% time horizon.

Key takeaway

For CTOs and VPs of Engineering evaluating AI automation strategies, the data suggests that increasing AI capabilities are not being offset by rising relative costs. You should confidently plan for broader AI deployment, as models can affordably complete tasks at 50% or 80% reliability horizons with cost ratios around 3% of human labor. This implies that even with retries for failures, automation remains profitable, and investing in higher inference compute for specific domains could further accelerate capability milestones.

Key insights

AI automation costs are not rising relative to human labor, enabling continued rapid capability expansion.

Principles

• AI cost ratios remain low at 50% reliability horizons.
• Longer tasks do not inherently incur higher AI cost ratios.
• Cost caps minimally impact AI capability doubling times.

Method

The study calculates cost ratio by estimating AI inference cost from token count (using OpenRouter pricing) and dividing by human task cost, analyzing trends at 50% and 80% reliability time horizons.

In practice

• Set low cost caps for AI tasks to maintain profitability.
• Focus on 50% or 80% reliability horizons for automation.
• Consider premium AI tiers for enhanced capabilities.

Topics

• AI Automation
• Inference Costs
• METR Time Horizons
• Cost Ratio Analysis
• AI Model Affordability

Code references

• AndersWoodruff/cost-ratio-graphing
• METR/eval-analysis-public

Best for: CTO, VP of Engineering/Data, Executive, AI Scientist, Director of AI/ML, Consultant

Related on AIssential

• 🔮 Exponential View #571: DeepSeek shows the future, again; drones on a learning curve; solar goes up, LLM pixels & tennis robots++ — Rapid iteration and cost-efficiency are critical for technological advantage in both warfare and AI development.
• AI is the Concorde of our time (Ep. 309) — Frontier AI's technical brilliance is overshadowed by its economically unsustainable, industrial-scale operational costs.
• My picture of the present in AI — AI is accelerating R&D and automating complex tasks, but faces alignment challenges and increasing cyber risks.
• AI Isn’t Replacing Humans As Fast As People Think — Because Intelligence Is Expensive — Scaling AI intelligence reliably in production incurs significant infrastructure and operational costs, slowing human job replacement.
• 📈 Data to start your week: Inside the AI boom – jobs, jargon & jittery uptime — AI is poised to automate junior roles, but widespread adoption faces cost and compute constraints.
• Why Your AI Costs Skyrocket As Your Product Gets Better — AI optimization paradoxically increases costs as product performance improves, challenging traditional SaaS economics.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Redwood Research blog.