How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study
Summary
A mixed-methods study, involving interviews with 20 practitioners from 12 organizations and a survey of 80 practitioners, investigates how developers build Software Engineering (SE) agents. These LLM-based agents understand large codebases and perform engineering tasks with limited human intervention. The research reveals that as implementation costs decrease, bottlenecks shift to non-coding activities like requirements, coordination, review, and deployment. A new central focus emerges on reviewing and evaluating agent output. The study characterizes a seven-stage workflow and identifies a move towards "evaluation-driven development", where evaluation guides iteration and specifications become versioned artifacts. It also details six challenges, including unreliable evaluation signals and "comprehension debt", along with adopted mitigation practices.
Key takeaway
For AI Engineers developing LLM-based Software Engineering agents, you should recognize that development bottlenecks are shifting from coding to non-coding aspects like requirements and coordination. Prioritize establishing robust evaluation frameworks for agent output, as this becomes a central and critical activity. Be prepared to manage "comprehension debt" and adapt to behavioral changes from provider-side model updates, integrating these considerations into your development workflow.
Key insights
Building SE agents shifts bottlenecks from coding to non-coding tasks and emphasizes "evaluation-driven development".
Principles
- Non-coding work becomes more visible in agent development.
- "Evaluation-driven development" steers agent iteration.
- Specifications should be versioned for human and agent use.
Method
Practitioners follow a seven-stage workflow for SE agent development, incorporating "evaluation-driven development" where specifications are versioned artifacts.
In practice
- Prioritize robust evaluation signals for agent output.
- Manage "comprehension debt" as codebases grow.
- Account for provider-side model update impacts.
Topics
- Software Engineering Agents
- LLM-based Agents
- Agent Development Workflow
- Evaluation-Driven Development
- Codebase Comprehension
- AI Engineering Challenges
Best for: Director of AI/ML, AI Architect, Research Scientist, AI Engineer, Machine Learning Engineer, AI Scientist
Related on AIssential
See Counsel's argued verdicts on the open AI decisions leaders are weighing →
Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.