Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Models

2026-04-29 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, short

Summary

A study benchmarked three frontier Large Language Models (LLMs) – Claude Opus 4.7, Claude Sonnet 4.6, and GPT-5.4 – on their ability to answer 100 natural-language questions over the Cleaned Contoso Retail Dataset in ClickHouse. The research specifically investigated the impact of providing LLMs with explicit business semantics, delivered via a 4 KB hand-authored markdown document, in addition to the database schema. Results showed that supplying this semantic layer significantly improved accuracy by +17 to +23 percentage points across all models. With the semantic document, all three models performed statistically indistinguishably, achieving 67.7-68.7% accuracy; without it, their accuracy ranged from 45.5-50.5%, also statistically indistinguishable. This indicates that explicit business semantics, rather than model choice within a tier, account for the significant variance in performance, by changing the nature of the task for the LLM.

Key takeaway

For AI Architects and NLP Engineers building natural-language interfaces for analytical databases, your primary focus should be on creating robust semantic layers. Providing explicit business semantics, such as a markdown document detailing measures and conventions, is far more impactful for improving LLM accuracy and reducing hallucinations than selecting a specific frontier model. Prioritize developing comprehensive semantic documentation to ensure reliable and accurate data analytics.

Key insights

Explicit business semantics significantly improve LLM accuracy and reduce hallucination in natural-language data querying.

Principles

• Schema alone is insufficient for reliable LLM data querying.
• Semantic layers suppress text-to-SQL errors structurally.
• Model choice is less critical than semantic context.

Method

Benchmarked three LLMs on 100 natural-language questions over a retail dataset, using a paired single-shot protocol to compare performance with and without a 4 KB semantic markdown document.

In practice

• Augment database schemas with semantic documentation.
• Use markdown for encoding business rules and conventions.
• Prioritize semantic context over model selection.

Topics

• LLM-Powered Data Analytics
• Semantic Layers
• Text-to-SQL
• Hallucination Mitigation
• Model Benchmarking

Best for: AI Architect, AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer, Data Scientist

Related on AIssential

• Semantic Foundations for Reliable Enterprise AI — Semantic clarity in data, achieved through robust data contracts and ontologies, is crucial for reliable enterprise AI outputs.
• BLINKG: A Benchmark for LLM-Integrated Knowledge Graph Generation — BLINKG provides a standardized benchmark to evaluate Large Language Models' effectiveness in mapping data schemes to ontology concepts for Knowledge Graph generation.
• Knowledge Engineering for Search and Content: A Practical Guide — Structured understanding of content and user intent is crucial for effective LLM-powered search and content experiences.
• The Ontology Engineering Challenge is Not Building It, but Mirroring the Delta — The core challenge in enterprise AI is correcting LLM's inherent concept understanding where it diverges from specific business definitions.
• Why machines that can write poetry still can’t agree on what “good” means — Ontology-based semantic layers are crucial for grounding agentic AI in explicit, governed business meaning, overcoming inherent linguistic ambiguity.
• Semantic Layers in the Wild: Lessons from Early Adopters — Semantic layers are becoming critical AI infrastructure, driven by the need for accurate, governed data for AI analytics.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.