SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution
Summary
EvoClaw is a novel benchmark designed to evaluate AI agents on continuous software evolution, addressing the limitations of existing benchmarks that focus on isolated coding tasks. It utilizes DeepCommit, an agentic pipeline that reconstructs verifiable Milestone DAGs from commit logs, representing semantically cohesive development goals. EvoClaw comprises 98 human-verified milestones across 7 open-source repositories and five programming languages, simulating real-world, dependency-constrained development. Evaluations of 12 frontier models across 4 agent frameworks revealed a significant performance drop: scores fell from >80% on isolated tasks to at most 38.03% (Claude Opus 4.6) in continuous settings, with a 13.37% Resolve Rate (Gemini 3 Pro). This highlights agents' profound struggle with long-term maintenance, error propagation, and technical debt accumulation.
Key takeaway
For AI Engineers deploying LLM-powered agents in continuous software development, recognize that current frontier models struggle profoundly with long-term code maintenance. While they achieve high scores on isolated tasks, their performance drops significantly in continuous settings due to accumulating technical debt and regressions. Prioritize agent designs that emphasize proactive codebase exploration and disciplined test verification to prevent error propagation, rather than solely focusing on feature implementation. Your evaluation metrics must balance new feature completeness with robust regression prevention.
Key insights
AI agents excel at isolated coding but struggle profoundly with continuous software evolution due to accumulating technical debt.
Principles
- Sustained software evolution demands maintaining system integrity and limiting error accumulation.
- Agent performance degrades significantly when early implementation decisions constrain subsequent ones.
- Proactive codebase exploration and disciplined test verification mitigate technical debt.
Method
DeepCommit reconstructs verifiable Milestone DAGs by preprocessing commit history, using LLM agents for milestone construction (seed discovery, consolidation, dependency inference, refinement), and resolving runtime environments for executability.
In practice
- Prioritize agentic exploration and verification tools in continuous development.
- Design evaluation metrics that balance new feature recall with regression prevention.
Topics
- AI Agents
- Software Evolution
- LLM Evaluation
- EvoClaw Benchmark
- DeepCommit Pipeline
- Technical Debt
Code references
Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer
Related on AIssential
See Counsel's argued verdicts on the open AI decisions leaders are weighing →
Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.