Near-Miss: Latent Policy Failure Detection in Agentic Workflows

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

The "Near-Miss" study introduces a novel metric for detecting latent policy failures in LLM-based agentic workflows, addressing a blind spot in current evaluation methods. Traditional approaches compare the final system state against ground truth, missing cases where agents bypass required policy checks but achieve correct outcomes due to favorable circumstances. These "near-misses" or "latent failures" are identified by analyzing agent trajectories, specifically assessing if tool-calling decisions were sufficiently informed. Building on the ToolGuard framework, which converts natural-language policies into executable guard code, the method was evaluated on the τ2-verified Airlines benchmark. Results show that latent failures occur in 8–17% of trajectories involving mutating tool calls, even when the final outcome is correct, highlighting the need to evaluate decision processes, not just final states.

Key takeaway

For MLOps Engineers evaluating LLM-based agentic workflows, relying solely on final outcome comparisons is insufficient. Your current evaluations likely miss "near-misses" where agents bypass policy checks but still achieve correct results. You should implement process-level policy adherence metrics, such as those analyzing agent decision-making and tool-calling informedness, to detect these latent failures and ensure robust, compliant automation.

Key insights

Latent policy failures in agentic workflows bypass checks but achieve correct outcomes, requiring process-level detection.

Principles

• Final outcome evaluation misses latent policy failures.
• Policy adherence needs process-level scrutiny.
• Informed tool-calling decisions are critical.

Method

The method uses the ToolGuard framework to convert natural-language policies into executable guard code, analyzing agent trajectories to assess if tool-calling decisions were sufficiently informed.

In practice

• Integrate process-level policy checks.
• Evaluate agent decision-making, not just outcomes.
• Use ToolGuard for policy enforcement.

Topics

• Agentic Workflows
• LLM Evaluation
• Policy Adherence
• Latent Failures
• ToolGuard Framework
• Business Process Automation

Best for: Research Scientist, AI Architect, AI Engineer, AI Scientist, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Your Agent Gave the Right Answer for the Wrong Reason — and You Have No Idea — Agentic AI systems require deep observability and multi-dimensional evaluation to prevent silent failures and ensure reliable, efficient operation.
• HANDBOOK.md: Can Agents Follow 100-Page Company Policies? — Frontier AI agents consistently fail to follow complex, multi-page company policies in real-world enterprise environments.
• Looking Is Not Picking: An Attention-Segment Account of Tool-Selection Failures in LLM Agents — LLM agent tool-selection failures stem from decision readout, not insufficient attention to the correct tool.
• When Behavioral Safety Evaluation Fails: A Representation-Level Perspective — Current LLM behavioral safety evaluations are insufficient, failing to detect deep representation-level vulnerabilities that an "audit gap" reveals.
• IBM and UC Berkeley Diagnose Why Enterprise Agents Fail Using IT-Bench and MAST — MAST diagnoses agentic LLM failures beyond success rates, revealing distinct failure signatures across model classes.
• Do More Agents Help? Controlled and Protocol-Aligned Evaluation of LLM Agent Workflows — BenchAgent reveals multi-agent LLM workflows often underperform single-agent setups under controlled conditions, except for specific runtime-generated designs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.