SchemaScope: How Join-Hop Depth Breaks Text-to-SQL in Large Language Models, and a Decomposition-Based Remedy

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

SchemaScope introduces a critical analysis of Large Language Models' (LLMs) performance on Text-to-SQL tasks, particularly concerning enterprise databases. The research identifies join-hop depth (h), the number of foreign key edges traversed, as a primary bottleneck. A new Join-Hop Depth (JHD) benchmark, comprising 410 human-annotated questions across 12 enterprise schemas, reveals a sharp accuracy decline: LLMs exceed 80% at h=1 but fall below 40% at h=4 and under 25% at h=6, typical for real analytics queries. To mitigate this, SchemaScope proposes a decomposition framework that breaks deep queries into sub-queries with h ≤ 2, executes them, and merges results. This method boosts execution accuracy from 46.8% to 67.3% on JHD (GPT-4o, h ≥ 3) and improves BIRD development set accuracy by +9.3 percentage points, primarily by eliminating "wrong join path" errors.

Key takeaway

For NLP Engineers or AI Scientists building Text-to-SQL solutions for enterprise databases, you must account for join-hop depth. Your current LLM solutions likely face a significant accuracy cliff beyond h=1. Implement a decomposition framework like SchemaScope to partition complex queries, improving execution accuracy by over 20 percentage points on deep queries and shifting error focus to aggregation and condition mistakes.

Key insights

LLMs struggle with high join-hop depth in Text-to-SQL, but a decomposition framework significantly improves accuracy.

Principles

• Join-hop depth (h) is a critical bottleneck for LLM Text-to-SQL accuracy.
• LLM performance sharply declines for queries with h > 1.
• Decomposition eliminates "wrong join path" errors in complex queries.

Method

SchemaScope decomposes deep Text-to-SQL queries into a sequence of sub-queries with h ≤ 2, executes them independently, and then merges the results.

In practice

• Benchmark LLMs using Join-Hop Depth (JHD) for enterprise database tasks.
• Apply query decomposition for Text-to-SQL queries exceeding h=2.
• Shift post-processing focus to condition and aggregation errors.

Topics

• Text-to-SQL
• Large Language Models
• Join-Hop Depth
• SchemaScope
• Query Decomposition
• Enterprise Databases
• LLM Benchmarking

Best for: Research Scientist, AI Engineer, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• DecoSearch: Complexity-Aware Routing and Plan-Level Repair for Text-to-SQL — DecoSearch improves Text-to-SQL accuracy for complex queries via complexity-aware routing and plan-level repair.
• DW-Bench: Benchmarking LLMs on Data Warehouse Graph Topology Reasoning — LLMs excel at orchestrating graph algorithms via tools but struggle with compositional multi-hop graph reasoning.
• Where to Place the Query? Unveiling and Mitigating Positional Bias in In-Context Learning for Diffusion LLMs via Decoding Dynamics — Query placement is a first-order variable in dLLM In-Context Learning, impacting performance as much as example selection.
• Semantic adapters in text-to-SQL for low-resource languages: the importance of semantic information — Injecting semantic structural knowledge into LLMs significantly boosts Text-to-SQL performance for low-resource languages.
• Unlocking the Power of Large Language Models for Multi-table Entity Matching — LLM4MEM uses LLMs to resolve semantic inconsistencies and efficiently match entities across multiple data tables.
• ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extraction — LLMs struggle with complex, enterprise-scale PDF-to-JSON extraction, especially as schema breadth increases.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.