IBM and UC Berkeley Diagnose Why Enterprise Agents Fail Using IT-Bench and MAST

2026-02-18 · Source: Hugging Face - Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure, Data Science & Analytics · Depth: Advanced, long

Summary

IBM Research and UC Berkeley collaborated to diagnose why agentic LLM systems fail in real-world IT automation tasks, such as incident triage and Kubernetes actions. They applied MAST (Multi-Agent System Failure Taxonomy) to analyze 310 ITBench SRE traces from Gemini-3-Flash, Kimi-K2, and GPT-OSS-120B models. Key findings indicate that frontier models like Gemini-3-Flash exhibit fewer, isolated failure modes (2.6/trace), often related to incorrect verification. In contrast, large open models like GPT-OSS-120B suffer from cascading failures (5.3/trace) due to reasoning mismatches and context poisoning. Kimi-K2 struggles with premature termination and unawareness of termination conditions, alongside significant action-reasoning mismatch. The study emphasizes that traditional success rate metrics are insufficient, advocating for detailed failure diagnostics to enable targeted engineering interventions.

Key takeaway

For AI Architects and Machine Learning Engineers building agents for enterprise IT workflows, understanding specific failure modes is crucial. Instead of relying solely on success rates, use diagnostic tools like MAST to identify "fatal" flaws such as incorrect verification (FM-3.3) or loss of conversation history (FM-1.4). This allows for targeted engineering solutions, like externalizing verification for Gemini-3-Flash or implementing state machines for Kimi-K2, leading to more robust and reliable agentic systems.

Key insights

MAST diagnoses agentic LLM failures beyond success rates, revealing distinct failure signatures across model classes.

Principles

• Externalize LLM verification.
• Control agent termination externally.
• Address input ambiguity explicitly.

Method

MAST converts unstructured execution logs into structured "failure vectors" across 14 patterns in three categories: System Design, Inter-Agent Misalignment, and Task Verification, enabling fine-grained failure analysis.

In practice

• Implement external verification gates.
• Use deterministic state machines for termination.
• Aggressively manage context hygiene.

Topics

• Multi-Agent Systems
• LLM Agents
• Failure Analysis
• IT Automation
• Benchmarking

Code references

• itbench-hub/ITBench
• multi-agent-systems-failure-taxonomy/MAST

Best for: AI Architect, Machine Learning Engineer, AI Scientist, MLOps Engineer, AI Engineer, Research Scientist

Related on AIssential

• Your AI Agent Is Battling Cancer — Biological pathologies offer a powerful diagnostic and treatment framework for AI agent system failures.
• A Multi-Dataset Benchmark for Evaluating LLM Agents in Microservice Failure Diagnosis — LLM agent evaluation for microservice diagnosis requires assessing reasoning processes, not just final answers, using multimodal, expert-validated benchmarks.
• Which Agent Causes Task Failures and When?Researchers from PSU and Duke explores automated failure attribution of LLM Multi-Agent Systems — Automated failure attribution for LLM Multi-Agent systems is a new, challenging problem requiring advanced reasoning.
• How Tool Chaining Fails in Production LLM Agents and How to Fix It — Tool chaining in LLM agents often fails silently in production due to cascading errors and context loss.
• Early Diagnosis of Wasted Computation in Multi-Agent LLM Systems via Failure-Aware Observability — Failure-aware observability diagnoses wasted computation in multi-agent LLM systems by mapping failures to online trace signals.
• POIROT: Interrogating Agents for Failure Detection in Multi-Agent Systems — POIROT enables LLM-MAS to self-diagnose failures by repurposing internal agents, outperforming external evaluators.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Hugging Face - Blog.