PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimization

2026-06-08 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, extended

Summary

PerfAgent is a profiler-guided, verifier-in-the-loop workflow designed to enhance large language model (LLM) agents for repository-level code optimization. It addresses three key LLM failure modes: missing real bottlenecks, premature termination after shallow speedups, and insufficient testing. PerfAgent feeds agents curated profiler summaries, employs an objective-driven loop controller that retains the fastest correct patch across up to 5 iterations, and uses selective validation via pytest-testmon to catch regressions efficiently. On two challenging optimization benchmarks, GSO and SWE-fficiency-Lite, PerfAgent more than doubles the rate of expert-matching patches over OpenHands with GPT-5.1, improving from 19.6% to 39.2% on GSO and from 26% to 74% on SWE-fficiency-Lite. It also surpasses an oracle best-of-five baseline at substantially lower cost.

Key takeaway

For AI Engineers tasked with optimizing large codebases using LLM agents, you should integrate profiler-guided, iterative refinement workflows like PerfAgent. This approach helps your agents move beyond superficial speedups and identify deep, cross-abstraction bottlenecks, leading to expert-level performance gains. By adopting selective testing and an objective-driven loop, you can ensure both significant speedups and robust correctness, avoiding costly regressions and premature termination.

Key insights

Profiler-guided, iterative feedback enables LLM agents to achieve expert-level repository-wide code optimization.

Principles

• Targeted profiling reveals hidden bottlenecks.
• Iterative refinement prevents premature optimization.
• Selective testing ensures correctness cheaply.

Method

PerfAgent uses py-spy for curated hotspot summaries, an objective-driven loop with up to 5 iterations to retain the fastest correct patch, and pytest-testmon for selective validation.

In practice

• Integrate py-spy for native-aware profiling.
• Implement iterative loops for performance goals.
• Use pytest-testmon for efficient regression checks.

Topics

• LLM Agents
• Code Optimization
• Performance Profiling
• Iterative Refinement
• Software Engineering Benchmarks
• GPT-5.1

Code references

• benfred/py-spy
• datacurve-ai/deep-swe
• openai/codex

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimization — PerfAgent uses profiler-guided, verifier-in-the-loop feedback to significantly improve LLM agent code optimization performance.
• Probe-and-Refine Tuning of Repository Guidance for Coding Agents — Iteratively refined, failure-informed guidance significantly improves LLM coding agent performance by boosting patch coverage.
• Asuka-Bench: Benchmarking Code Agents on Underspecified User Intent and Multi-Round Refinement — Real-world code generation demands iterative refinement from underspecified user intent, a capability Asuka-Bench effectively measures.
• ATLAS: Agentic Test-time Learning-to-Allocate Scaling — ATLAS enables LLM orchestrators to dynamically manage test-time scaling for improved reasoning and efficiency.
• PhoenixRepair: Rethinking Repair Strategy Exploration in Software Agents — PhoenixRepair enhances software agent repair by systematically exploring multiple edit locations and iteratively refining patch generation.
• Databricks Benchmarked Coding Agents on Its Own Codebase. The Results Should Change How You Buy — Internal, polyglot codebase benchmarks reveal true coding agent performance and cost-effectiveness beyond public metrics.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.