Tool-Aware Planning for Contact-Center Analytics: Evaluating LLMs through Lineage-Guided Query Decomposition

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

A new domain-grounded benchmark and evaluation framework has been introduced for tool-aware plan generation in contact-center analytics. This framework assesses how Large Language Models (LLMs) decompose business queries into executable steps using structured tools (Text2SQL over Snowflake), unstructured tools (RAG over transcripts), and LLM-based synthesis, incorporating explicit `depends_on` relations for parallel execution. The contributions include a reference-based plan evaluation framework with seven metric dimensions and a one-shot evaluator, alongside a lineage-driven data curation methodology. A large-scale study of 14 LLMs revealed persistent struggles with compound queries and plans exceeding four steps. The highest aggregate metric-wise score was 84.8 (Claude-3-7-Sonnet), while the strongest one-shot A+ rate was only 49.75% (o3-mini). Lineage provided mixed overall gains but improved several strong models and step executability, exposing weaknesses in tool understanding and completeness.

Key takeaway

For NLP Engineers building agentic LLM systems for enterprise question-answering, particularly in contact centers, this research highlights critical limitations. Your LLMs will likely struggle with complex, multi-step queries and plans longer than four steps, even with lineage guidance. Focus initial deployments on simpler analytical tasks and invest in robust, multi-dimensional evaluation frameworks to identify and mitigate persistent weaknesses in tool understanding and prompt alignment before scaling.

Key insights

LLMs struggle with complex, multi-step tool-aware planning for contact-center analytics, despite evaluation frameworks and lineage guidance.

Principles

• Tool-aware planning requires explicit `depends_on` relations.
• Iterative evaluation refines plan lineages effectively.
• Shorter, simpler plans are easier for LLMs to generate.

Method

A lineage-driven data curation methodology uses an iterative evaluator-optimizer loop to refine initial plans into high-quality plan lineages, reducing manual effort in benchmark creation.

In practice

• Evaluate LLM planning with seven metric dimensions.
• Incorporate lineage to improve plan executability.
• Prioritize simpler, shorter plans for LLM agents.

Topics

• Tool-Aware Planning
• LLM Evaluation
• Contact Center Analytics
• Query Decomposition
• Plan Lineage
• Agentic AI

Best for: Research Scientist, AI Architect, AI Engineer, AI Scientist, NLP Engineer, Director of AI/ML

Related on AIssential

• QueryWeaver: Reliable Multi-Tool Query Execution Planning via LLM-Based Graph Generation — QueryWeaver transforms natural language into structured graphs for reliable multi-tool query execution using a deterministic, dependency-resolving planner.
• Machines Have Been Trying to Understand Us for Decades. They’re Finally Getting Close. — The transformer architecture, combined with massive scale, fundamentally reshaped NLP, enabling emergent language understanding capabilities previously deemed impossible.
• Benchmarking Local LLMs for Natural-Language-to-SQL Querying in Biopharmaceutical Manufacturing: An Empirical Benchmark on Consumer-Grade Hardware — Code-tuned general-purpose LLMs outperform domain-specific medical LLMs for T-SQL generation on manufacturing schemas.
• TopBench: A Benchmark for Implicit Predictive Reasoning in Tabular Question Answering — LLMs struggle with implicit predictive reasoning in tabular data, often misinterpreting intent as simple retrieval.
• To Call or Not to Call: A Framework to Assess and Optimize LLM Tool Calling — LLM tool-calling decisions, especially for web search, often misalign with true necessity and utility.
• SING: Synthetic Intention Graph for Scalable Active Tool Discovery in LLM Agents — Intention-aware graph structures enable accurate, context-efficient tool discovery for LLM agents in large ecosystems.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.