Every Developer Thinks AI Makes Them 20% Faster. Measured, They’re 19% Slower.

2026-06-22 · Source: Towards AI - Medium · Field: Technology & Digital — Software Development & Engineering, Artificial Intelligence & Machine Learning · Depth: Intermediate, quick

Summary

An independent research group, METR, conducted a study involving 16 senior developers to assess the actual impact of AI coding tools on productivity. Participants were tasked with fixing issues in familiar codebases, with half their sessions utilizing AI tools and half without. Developers expected a 24% speedup with AI and, even after the experiment, believed they were 20% faster. However, objective measurements revealed that developers were actually 19% slower when using AI tools. This significant discrepancy highlights a cognitive bias where perceived productivity gains from AI coding assistants do not align with measured performance.

Key takeaway

For engineering managers evaluating AI coding tools, you must implement objective productivity metrics rather than relying on developer self-assessments. Your teams may perceive speedups while actually incurring slowdowns, leading to hidden costs and misallocated resources. Prioritize rigorous A/B testing for AI tool adoption to ensure genuine efficiency gains and avoid productivity traps.

Key insights

Developers using AI coding tools perceived a 20-24% speedup, but were actually 19% slower in a controlled METR study.

Principles

• Perceived AI productivity often exceeds actual measured gains.
• Senior developers may experience greater slowdowns with AI tools.

Method

METR's experiment involved 16 senior developers fixing issues in familiar codebases, randomizing tasks between AI-assisted and non-AI sessions.

In practice

• Cognitive bias can lead to overestimating AI tool benefits.
• AI tool integration may incur hidden costs for teams.

Topics

• Developer Productivity
• AI Coding Tools
• Productivity Measurement
• Cognitive Bias
• Software Engineering
• METR Study

Best for: CTO, VP of Engineering/Data, Executive, Software Engineer, Director of AI/ML, Research Scientist

Related on AIssential

• It’s starting… — AI systems are increasingly automating their own development, accelerating progress while shifting human roles to high-level judgment.
• The Arithmetic of Productivity Boosts: Why Does a “40% Increase in Productivity” Never Actually Work? — Marketing often misrepresents specific task productivity gains as overall productivity improvements.
• AI Coding Assistants Haven’t Sped up Delivery Because Coding Was Never the Bottleneck — AI coding tools shift bottlenecks from coding to specification and verification, demanding new team structures and workflows.
• So where are all the AI apps? — The "AI effect" on software productivity is concentrated in popular AI-related packages, not a general industry boost.
• Building OpenCode with Dax Raad — AI coding tools don't inherently accelerate software development; engineering judgment and strategic thinking remain critical.
• What the Top 1% of Engineering Teams Do Differently with AI — Top engineering teams achieve exponential productivity by integrating AI into flatter structures and optimizing development workflows.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards AI - Medium.