DW-Bench: Benchmarking LLMs on Data Warehouse Graph Topology Reasoning

2026-04-22 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

DW-Bench is a new benchmark designed to evaluate Large Language Models (LLMs) on their ability to perform graph topology reasoning over data warehouse schemas. Unlike traditional Text-to-SQL benchmarks, DW-Bench focuses on tasks like tracing foreign key paths, detecting disconnected schema silos, and propagating impact through ETL lineage chains. It comprises 1,046 questions across 13 subtypes, three difficulty levels, and five datasets, including real-world and synthetic schemas. The benchmark evaluates six baselines, ranging from flat context injection to tool-calling and code execution, using Gemini 2.5 Flash, DeepSeek-V3, and Qwen2.5-72B. Results indicate that tool-augmented baselines achieve 87–90% micro-EM, significantly outperforming static methods (63–81%). A key finding is that LLMs struggle with compositional multi-hop tasks, particularly the "combined_impact" subtype, where no method exceeds 61% EM, highlighting a structural reasoning ceiling.

Key takeaway

For AI engineers developing LLM applications for data warehouse management, you should prioritize integrating robust graph reasoning tools. While LLMs show strong performance on single-hop and simple multi-hop queries when augmented with tools, their ability to perform complex, compositional graph reasoning, such as chaining lineage traversal with foreign key expansion, remains a significant challenge. Focus on developing hybrid architectures that combine interactive tools with learned graph representations to bridge this compositional reasoning gap.

Key insights

LLMs excel at orchestrating graph algorithms via tools but struggle with compositional multi-hop graph reasoning.

Principles

• Tool-use significantly enhances LLM graph reasoning.
• Lexical cues can mask true structural understanding.
• Compositional reasoning is a primary LLM bottleneck.

Method

DW-Bench generates questions deterministically from graph structures using NetworkX, categorizing them by structural complexity (easy, medium, hard) and employing an obfuscation protocol to randomize table names.

In practice

• Use tool-augmented LLMs for schema topology tasks.
• Prioritize compositional reasoning in LLM development.
• Evaluate LLMs with obfuscated schemas to test true understanding.

Topics

• DW-Bench
• LLM Benchmarking
• Data Warehouse Schemas
• Graph Topology Reasoning
• Tool-Augmented LLMs

Code references

• AJamal27891/dw-bench

Best for: AI Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• TopoBench: Benchmarking LLMs on Hard Topological Reasoning — LLMs struggle with topological reasoning due to difficulty extracting spatial constraints, not inherent reasoning limitations.
• Beyond Vector Similarity: A Structural Analysis of Graph-Augmented Retrieval for Industrial Knowledge Graphs — Effective LLM-based graph reasoning hinges on providing robust computational operators, not just enhanced model intelligence.
• Are Large Language Models Suitable for Graph Computation? Progress and Prospects — LLMs excel as graph computation planners but struggle as direct executors for large, exact tasks.
• Integrating Graphs, Large Language Models, and Agents: Reasoning and Retrieval — Integrating graphs with LLMs enhances reasoning, retrieval, and decision-making by combining structured knowledge with semantic understanding.
• GTBench: A Curriculum-Grounded Benchmark for Evaluating LLMs as Mathematical Research Assistants in Graph Theory — LLMs struggle with mathematical reasoning, particularly graduate-level proofs, with GPT-5 significantly outperforming other models.
• Code-on-Graph: Iterative Programmatic Reasoning via Large Language Models on Knowledge Graphs — Code-on-Graph integrates LLMs with KGs using programmatic reasoning, representing schemas as Python classes for flexible, scalable knowledge utilization.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.