What Do AI Agents Actually Change? An Empirical Taxonomy of Mutation Patterns in Performance-Improving Pull Requests

2026-07-08 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, medium

Summary

An empirical study classified 1,254 performance-relevant diff hunks from 216 pull requests (PRs) generated by five AI coding agents (Devin, GitHub Copilot, Cursor, OpenAI Codex, and Claude Code) within the AIDev-pop dataset. This analysis, using an 18-category syntactic mutation taxonomy and a dual-LLM intersection pipeline, revealed that "name_modification" (37.0%), "object_creation" (26.4%), and "type_change" (22.7%) are the dominant mutation patterns in performance-improving PRs. This profile significantly differs from prior Genetic Improvement (GI) corpora, where "no_change" accounted for 84%. The research also established that each agent system exhibits a unique mutation vocabulary, and specific performance strategies activate distinct subsets of mutation categories. These findings suggest that agent identity and target strategy serve as valuable priors for refining the operator space in Search-Based Software Engineering (SBSE).

Key takeaway

If you are developing or refining automated code optimization tools using AI agents, you should incorporate agent identity and target performance strategy as priors. This narrows the effective mutation operator space from 18 categories to roughly five, improving efficiency. Be aware that agent profiles co-vary with language ecosystems, impacting generalizability. Tailor your mutation operators based on the specific agent and optimization goal.

Key insights

AI agent performance PRs show distinct mutation patterns, unlike human or general GI code.

Principles

• Agent identity shapes mutation patterns.
• Performance strategy dictates mutation subsets.
• Prior GI taxonomies don't transfer directly.

Method

A dual-LLM intersection pipeline classified 1,254 diff hunks from 216 PRs using an 18-category syntactic mutation taxonomy, with claude-sonnet-4-6 and gpt-5.4 as judges.

In practice

• Use agent/strategy as SBSE operator priors.
• Focus on "name_modification" for dependency swaps.
• Consider "type_change" for data structure upgrades.

Topics

• AI Coding Agents
• Genetic Improvement
• Search-Based Software Engineering
• Code Optimization
• Mutation Analysis
• Pull Requests

Code references

• 5uper6rain/ssbse-challenge-2026

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Software Engineer

Related on AIssential

• Shopify/liquid: Performance: 53% faster parse+render, 61% fewer allocations — AI-driven "autoresearch" can yield significant performance gains even in mature, highly optimized codebases.
• Coding Agents Meet Data Science — AI coding agents boost developer productivity but introduce new bottlenecks and require specialized adaptation for data science workflows.
• What Do Evolutionary Coding Agents Evolve? — Evolutionary coding agents often re-introduce deleted code and achieve gains through hyperparameter tuning, not just novel algorithmic structures.
• Superficial Success vs. Internal Breakdown: An Empirical Study of Generalization in Adaptive Multi-Agent Systems — Adaptive multi-agent systems exhibit topological overfitting and illusory coordination, limiting their generalization and practical utility.
• Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Study — AI IDE rules, though often low-level, significantly improve software artifact compliance when updated.
• AgentGA: Evolving Code Solutions in Agent-Seed Space — AgentGA optimizes autonomous code generation by evolving agent seeds, improving performance through inherited artifacts.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.