Structured Feedback Improves Repair in an LLM Agent Loop

2026-07-15 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

VeriHarness, a novel code-controlled agent loop, enhances the repair capabilities of LLM agents by providing structured feedback during retry attempts. This system allows models to generate candidates while external validators manage acceptance, budgets, and traces. Researchers compared raw diagnostic feedback against structured feedback that precisely identifies the failure location, observed value, and admissible alternatives. Across 50 paired TextWorld games, under a four-call cap, feedback incorporating all three fields dramatically improved terminal success. For Qwen2.5-Coder-14B, success rose from 14/50 to 36/50 (+44 percentage points), and for Llama-3.1-8B, it increased from 8/50 to 29/50 (+42 points). Ablation studies revealed that the "admissible alternatives" component contributed most to these gains, with location and observed value alone showing minimal improvement. Interestingly, presenting complete repair information in prose yielded similar success rates to JSON, indicating the syntax itself is not the primary factor.

Key takeaway

For AI Engineers building LLM agents that require self-correction, you should integrate structured feedback mechanisms into your agent loops. Focus on providing explicit "admissible alternatives" alongside failure location and observed values, as this component drives significant repair success. This approach can boost terminal success rates by over 40 percentage points, regardless of whether you use JSON or prose for feedback. Prioritize content clarity over specific syntax to enhance agent reliability.

Key insights

Structured feedback, particularly with admissible alternatives, substantially improves LLM agent repair success rates.

Principles

• Explicitly identifying admissible alternatives is key to effective LLM agent repair.
• The content of repair feedback matters more than its specific syntax (e.g., JSON vs. prose).
• External validation controlling acceptance and budgets enhances agent loop robustness.

Method

VeriHarness operates as a code-controlled agent loop where external validators manage acceptance, budgets, and traces, providing feedback on failure location, observed value, and admissible alternatives.

In practice

• Design agent loops to provide explicit "admissible alternatives" in feedback for repair.
• Prioritize clear content in feedback over specific serialization formats like JSON.

Topics

• LLM Agents
• Structured Feedback
• Agent Repair
• VeriHarness
• Qwen2.5-Coder-14B
• Llama-3.1-8B

Best for: Research Scientist, AI Architect, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Agent Control Patterns — Part 2: Reflection — A Simple Way to Improve Answer Quality — Separating generation from evaluation through reflection improves AI answer quality without model retraining.
• Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair — Agentic code repair loops lack inherent reliability; verifiable state-bound contracts are crucial for trustworthy revisions.
• Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agents — Failed trajectories offer rich, untapped information for improving computer-use agents without retraining.
• Unlocking LLM Code Correction with Iterative Feedback Loops — Iterative feedback loops significantly enhance LLM code correction, especially for reasoning models and specific error types.
• Self-Reflective APIs: Structure Beats Verbosity for AI Agent Recovery — Structured, machine-readable API feedback significantly improves AI agent error recovery and token efficiency for certain LLMs.
• PhoenixRepair: Rethinking Repair Strategy Exploration in Software Agents — PhoenixRepair enhances software agent repair by systematically exploring multiple edit locations and iteratively refining patch generation.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.