How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study

2026-07-12 · Source: Artificial Intelligence · Field: Technology & Digital — Software Development & Engineering, Artificial Intelligence & Machine Learning · Depth: Intermediate, quick

Summary

A mixed-methods study, involving semi-structured interviews with 20 practitioners from 12 organizations and an online survey of 80 practitioners, investigates how developers build Software Engineering (SE) agents. This research reveals that as implementation costs decrease, bottlenecks shift from coding to non-coding activities like requirements, coordination, review, and deployment. A new central challenge is reviewing and evaluating agent output. The study characterizes a seven-stage workflow and highlights a move towards evaluation-driven development, where evaluation guides iteration and specifications become versioned artifacts for both humans and agents. Furthermore, it identifies six key challenges, including unreliable evaluation signals, "comprehension debt" as code outpaces understanding, and behavioral changes caused by provider-side model updates.

Key takeaway

For Software Engineering teams developing LLM-based agents, you should anticipate a shift in development bottlenecks from coding to non-coding activities like requirements and coordination. Prioritize robust evaluation strategies for agent output early in your process, adopting an evaluation-driven development approach. Be prepared to manage "comprehension debt" and account for behavioral changes from provider-side model updates to ensure agent reliability.

Key insights

Building SE agents shifts development bottlenecks to non-coding tasks and elevates agent output evaluation to a central role.

Principles

• SE agent development shifts bottlenecks.
• Evaluation drives agent iteration.
• Specifications are versioned artifacts.

Method

Practitioners adopt a seven-stage workflow for SE agent development, characterized by evaluation-driven iteration where specifications are versioned artifacts read by both humans and agents.

In practice

• Prioritize robust evaluation signals.
• Actively manage comprehension debt.
• Monitor model update impacts.

Topics

• Software Engineering Agents
• LLM Agents
• Agent Development Workflow
• Evaluation-Driven Development
• Comprehension Debt
• Model Behavioral Changes

Best for: AI Architect, CTO, VP of Engineering/Data, AI Engineer, Machine Learning Engineer, Software Engineer

Related on AIssential

• How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study — Building SE agents shifts bottlenecks from coding to non-coding tasks and emphasizes "evaluation-driven development".
• Faster Code, Slower Understanding: A Survival Note from a Year of Agent-First Engineering — AI shifts the engineering bottleneck from code production to comprehension and maintenance, demanding new optimization strategies.
• Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios — LLM agents struggle with end-to-end software generation, often preferring monolithic designs over modularity.
• Components of a Coding Agent: How LLMs Become Software Engineers — Coding agents transform LLMs into dynamic, multi-step problem solvers through structured system design.
• Thoughts on coding agents — Coding agents fundamentally shift software engineering workflows, demanding explicit context provision and deliberate model-building from developers.
• Buffett Had Moody’s Manuals. I Built a Desk of AI Analysts. — Building trustworthy multi-agent AI systems requires rigorous engineering, particularly robust validation, retry logic, and honest evaluation.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.