Who Broke the System? Failure Localization in LLM-Based Multi-Agent Systems

2026-07-11 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

AgentLocate is a new framework designed for failure localization in LLM-based multi-agent systems, addressing the challenge of identifying the responsible agent and the earliest decisive error step in complex, long-horizon interactions. It employs an LLM-based "Judge" for initial hypothesis generation, followed by a "Judge-Evaluator refinement" stage where multiple independent "Evaluators" with diverse reasoning styles verify the hypothesis. Their assessments are aggregated using a confidence-aware voting scheme, and this feedback is used to fine-tune the Judge via parameter-efficient fine-tuning. Evaluated on Who&When and Aegis-Bench benchmarks, AgentLocate consistently outperforms existing methods, achieving over 50% average agent-level accuracy. It also demonstrates efficiency, running over 20 times faster than AgenTracer (315 vs. 6,752 seconds) at a fraction of the cost (\$0.148 vs. \$3.204) using Qwen-7B. Optimal performance is achieved with three evaluators and a single refinement round.

Key takeaway

For MLOps Engineers or AI Scientists debugging multi-agent LLM systems, AgentLocate offers a robust solution for pinpointing failure origins. You should integrate this framework to accurately identify the responsible agent and decisive error step, significantly streamlining debugging workflows. Its efficiency and superior accuracy over existing methods mean you can diagnose complex system failures faster and more cost-effectively, improving overall system reliability.

Key insights

AgentLocate reliably pinpoints failure origins in multi-agent LLM systems through iterative, verified, and refined attribution.

Principles

• Localization benefits from multi-perspective verification.
• Feedback-driven refinement improves attribution fidelity.
• Earliest decisive step is key for causal responsibility.

Method

An LLM Judge generates a failure hypothesis, independent Evaluators verify it with confidence-weighted voting, and their feedback fine-tunes the Judge for improved accuracy.

In practice

• Use three evaluators for optimal performance.
• Apply parameter-efficient fine-tuning (LoRA) for Judge adaptation.

Topics

• LLM Multi-Agent Systems
• Failure Localization
• Debugging AI Systems
• AgentLocate Framework
• LLM-as-a-Judge
• Parameter-Efficient Fine-Tuning

Best for: Research Scientist, AI Scientist, MLOps Engineer, AI Engineer

Related on AIssential

• Who Broke the System? Failure Localization in LLM-Based Multi-Agent Systems — AgentLocate attributes failures in LLM multi-agent systems to specific agents and decisive steps using LLM judging and multi-perspective verification.
• FALAT: Tracing Failures in LLM Agent Trajectories via Dependency-Guided Search — Failure attribution in LLM agent trajectories demands dependency-guided search to distinguish root causes from propagated errors.
• Your Agent Loop Is Fine. Your Tools Are Why It Breaks. — Agent reliability hinges on well-defined tools, not complex orchestration loops.
• The Invisible Failure Mode of Agentic AI — Agentic AI failures are often invisible resource issues, not reasoning flaws, requiring real-time, local observability.
• AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP — LLM agents require controlled fault injection and fix verification to ensure robustness against real-world tool failures.
• AI Agents Don't Fail Because of the LLM. They Fail Because of the System Around It. — AI agent failures are primarily due to system design, not LLM limitations.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.