What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents

2026-07-08 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Robotics & Autonomous Systems · Depth: Expert, medium

Summary

TraceProbe is a trajectory-diagnostic framework designed to analyze the process evidence of LLM-powered coding agents, moving beyond the limited "resolve rate" metric. It normalizes raw agent runs into a canonical nine-type action taxonomy with deterministic effect labels. The framework includes two rule-based modules: Insight, which identifies single-trajectory anti-patterns like search loops and verification skips, and Converge, which aligns pairs of runs to classify behavioral divergences. Applied to 2,500 trajectories from five production settings on SWE-Bench Verified, involving three scaffolds (Claude Code, Codex, OpenCode) and three model backbones (Opus 4.6, GPT-5.4, GLM-5), TraceProbe found that function selection and completion behavior localize failure more effectively than file choice. It also revealed that even resolved runs differ in efficiency and incurred failed work.

Key takeaway

For MLOps Engineers evaluating coding agents, relying solely on resolve rate provides an incomplete picture of agent efficiency and robustness. You should integrate trajectory analysis tools like TraceProbe to understand how agents achieve outcomes, identifying avoidable effort and specific failure patterns. This allows you to make informed decisions on agent deployment and targeted improvements, comparing process distributions across different agent settings.

Key insights

TraceProbe diagnoses coding agent behavior by normalizing trajectories into canonical actions and detecting anti-patterns or divergences.

Principles

• Resolve rate alone hides critical process differences in coding agent performance.
• Trajectory analysis requires a normalized representation for scalable comparison.
• Rule-based detectors provide auditable, repeatable diagnostic signals for agent behavior.

Method

TraceProbe normalizes raw trajectories into a nine-type action taxonomy with deterministic effect labels. It then uses an Insight module for single-trajectory anti-patterns and a Converge module for aligning and classifying divergences between run pairs.

In practice

• Use TraceProbe to compare agent processes across different scaffolds or models.
• Identify specific anti-patterns like search loops in agent trajectories.
• Localize failure evidence to function selection rather than just file choice.

Topics

• Coding Agents
• Trajectory Analysis
• LLM Evaluation
• SWE-Bench
• Debugging Anti-patterns
• MLOps

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Agent trajectories as programs: fingerprinting and programming coding-agent behavior — Agent problem-solving trajectories can be fingerprinted and analyzed procedurally to reveal distinct behavioral patterns beyond mere success rates.
• TRACE: Trajectory Risk-Aware Compression for Long-Horizon Agent Safety — TRACE uses a Compressor-Reader design to compress long LLM agent trajectories into a latent evidence state for improved safety detection.
• CodeTracer: Towards Traceable Agent States — CodeTracer provides structured tracing and failure localization for complex code agent trajectories, improving debugging and recovery.
• Measuring What Matters with Jules — Proactive AI coding agents require goal-oriented evaluation focused on insight quality and exploration, not just task completion.
• Mastering Agentic Techniques: AI Agent Evaluation — AI agent evaluation assesses end-to-end system behavior in dynamic environments, distinct from static model capability benchmarks.
• FALAT: Tracing Failures in LLM Agent Trajectories via Dependency-Guided Search — Failure attribution in LLM agent trajectories demands dependency-guided search to distinguish root causes from propagated errors.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.