To Run or Not to Run: Analyzing the Cost-Effectiveness of Code Execution in LLM-Based Program Repair

2026-01-23 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, extended

Summary

An empirical study analyzed the cost-effectiveness of code execution in LLM-based program repair, examining 7,745 agent traces from SWE-bench leaderboard submissions and 3,000 controlled repair attempts across 200 SWE-bench instances. Using agents like Claude Code, Codex, and open-source OpenCode with Qwen2.5-Coder-32B, the study evaluated four execution paradigms. Findings indicate agents average 8.8 test runs per task, with late-stage executions achieving 57.9% success. Crucially, execution restrictions had minimal impact on repair success: the resolve-rate gap between "Prohibited" and "Unrestricted" modes was only 1.25pp for commercial agents (not statistically significant, p>0.05) and ≈ 0pp for OpenCode. "Prohibited" mode saved 56–62% tokens and 48–54% wall-clock time for Claude Code, suggesting execution is often applied indiscriminately.

Key takeaway

For ML engineers designing or deploying LLM-based program repair agents, recognize that unrestricted code execution often provides minimal marginal benefit while incurring substantial token and wall-clock costs. Prioritize adaptive execution strategies that selectively engage testing only when high-value information gain is expected, rather than treating it as a default capability. This approach can significantly reduce operational expenses and environment setup overhead without compromising repair effectiveness.

Key insights

Code execution in LLM-based program repair often provides minimal benefit despite significant costs, suggesting indiscriminate application.

Principles

• Code execution is a resource with explicit cost-benefit.
• Late-stage executions yield higher success rates.
• Partial execution access can be counterproductive.

Method

A two-stage empirical study analyzed 7,745 agent traces and conducted 3,000 controlled repair attempts across 200 SWE-bench instances under varying execution access paradigms.

In practice

• Restrict execution for cost-sensitive LLM agents.
• Adopt execution selectively, based on clear value.
• Focus on enhancing execution feedback quality.

Topics

• LLM Agents
• Program Repair
• Code Execution
• Cost-Effectiveness
• SWE-bench
• Resource Management

Code references

• mathieu0905/To_Run_Or_Not_To_Run
• opencode-ai/opencode

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, Software Engineer

Related on AIssential

• Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming — LLMs can actively verify and debug competitive programming code, but current inference costs limit practical deployment.
• Evaluating Agentic Configuration Repair for Computer Networks — Agentic LLM architectures significantly improve network configuration repair efficacy and safety through iterative validation.
• PracRepair: LLM-Empowered Automated Program Repair Inspired by Human-Like Debugging Practices — PracRepair enhances LLM-based automated program repair by integrating dynamic execution and validation feedback, mimicking human debugging.
• Atropos: Improving Cost-Benefit Trade-off of LLM-based Agents under Self-Consistency with Early Termination and Model Hotswap — Atropos improves LLM agent cost-benefit by predicting inference failure and hotswapping to a more capable model.
• Enhancing Software Engineering Through Closed-Loop Memory Optimization — MemOp grounds memory utility in validated downstream impact to optimize LLM-based SE agents.
• Are Tools All We Need? Unveiling the Tool-Use Tax in LLM Agents — Tool augmentation in LLMs incurs a "tool-use tax" that often negates benefits, especially with semantic noise.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.