How Do AI Coding Agents Contribute to Software Development? an Empirical Study of Agentic Pull Requests
Summary
An empirical study analyzed 220,612 closed pull requests (PRs) from 489 popular Python repositories, including 9,428 agent-generated PRs from OpenAI Codex, GitHub Copilot, Claude Code, Cursor, and Devin. The study characterized agentic PRs against human-generated ones across the software development lifecycle. Findings indicate Claude Code PRs have the highest merge rate (84.3%), while Devin's are lowest (43%). Agentic PR merge probability slightly decreases in the second quarter (63.7%) before peaking in the third (68.2%). AI agents are predominantly used for documentation, dependency management, and testing tasks, with operational and additive changes showing higher merge ratios. Agentic PRs generally exhibit comparable or lower defect proneness than human PRs, with most complexity differences being practically negligible.
Key takeaway
If you are a Director of AI/ML evaluating coding agent adoption, recognize that agents like Claude Code demonstrate higher PR merge rates, particularly for structured tasks such as documentation or dependency management. While agentic PRs show comparable or lower defect proneness, their complexity metrics are often negligibly different from human contributions. You should focus agent deployment on well-defined tasks and monitor agent-specific performance to optimize integration and maintain software quality.
Key insights
AI coding agents show varied merge rates and task specializations, often introducing fewer defects than human contributions.
Principles
- Agent mergeability varies significantly by agent and task type.
- Operational and additive changes yield higher agentic PR acceptance.
- Agentic PRs generally have lower defect proneness.
Method
The study used topic modeling (BERTopic with Gemini-embedding-001, UMAP, K-means) to categorize 9,428 agentic PRs into 24 development tasks, then compared merge rates and defect proneness using GLMM and SZZ.
In practice
- Delegate documentation and dependency updates to agents.
- Exercise caution with LLM integration or JSON template tasks.
- Monitor agent performance per agent and task type.
Topics
- AI Coding Agents
- Pull Request Analysis
- Software Development Lifecycle
- Code Quality
- Defect Proneness
- Large Language Models
Code references
Best for: AI Engineer, Machine Learning Engineer, AI Scientist, Research Scientist, Software Engineer, Director of AI/ML
Related on AIssential
See Counsel's argued verdicts on the open AI decisions leaders are weighing →
Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.