IndicDB -- Benchmarking Multilingual Text-to-SQL Capabilities in Indian Languages

2026-04-15 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

IndicDB is a new multilingual Text-to-SQL benchmark designed to evaluate cross-lingual semantic parsing in diverse Indic languages, addressing a gap in existing benchmarks that primarily focus on Western contexts and simplified schemas. The benchmark utilizes relational schemas derived from real-world administrative data on open-data platforms like the National Data and Analytics Platform (NDAP) and the India Data Portal (IDP). IndicDB features 20 databases across 237 tables, with an average of 11.85 tables per database and join depths up to six. An iterative three-agent framework (Architect, Auditor, Refiner) was used to convert denormalized government data into rich relational structures. The benchmark includes 15,617 tasks in English, Hindi, and five other Indic languages. Evaluations of models such as DeepSeek v3.2, MiniMax 2.7, LLaMA 3.3, and Qwen3 revealed a 9.00% performance drop from English to Indic languages, termed the "Indic Gap," attributed to harder schema linking, increased structural ambiguity, and limited external knowledge.

Key takeaway

For research scientists developing or fine-tuning Large Language Models for Text-to-SQL applications, you should prioritize evaluating your models against multilingual benchmarks like IndicDB. The identified "Indic Gap" highlights the need to specifically address challenges in schema linking, structural ambiguity, and external knowledge for non-Western languages. Focusing on these areas will improve real-world performance and applicability in diverse linguistic contexts.

Key insights

IndicDB benchmarks multilingual Text-to-SQL for Indic languages, revealing a significant performance gap compared to English.

Principles

• Real-world data increases benchmark complexity.
• Multilingual LLM performance varies by language.
• Schema linking is critical for Text-to-SQL.

Method

An iterative three-agent framework (Architect, Auditor, Refiner) converts denormalized government data into rich relational schemas, ensuring structural rigor and high relational density for Text-to-SQL benchmarks.

In practice

• Evaluate LLMs on IndicDB for Text-to-SQL.
• Focus on schema linking for Indic languages.
• Consider data complexity in benchmark design.

Topics

• Multilingual Text-to-SQL
• Indic Languages
• LLM Benchmarking
• Semantic Parsing
• Relational Databases

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• IndicDB -- Benchmarking Multilingual Text-to-SQL Capabilities in Indian Languages — IndicDB reveals a significant "Indic Gap" in LLM Text-to-SQL performance for Indian languages compared to English.
• To Describe or Not to Describe? Benchmarking Database Representations for Schema Linking in Text-to-SQL — LLM-generated natural language descriptions enhance Text-to-SQL schema linking more effectively than DDL/DML.
• UrduMMLU: A Massive Multitask Benchmark for Urdu Language Understanding — Native, culturally-grounded benchmarks reveal significant LLM knowledge gaps in non-English, humanities-focused content.
• Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets — Automated framework improves multilingual LLM benchmark translation quality using USI and T-RANK.
• Released a free 9.8M doc Indic multilingual corpus — Hindi, Bengali, Tamil, Telugu + 7 more (CC0, HuggingFace) [P] — A new 9.8M document, 11-language Indic multilingual corpus is now freely available under CC0.
• Beyond Bilingual Transfer: Multilingual Code-Switching in Instruction Tuning — Multilingual code-switching instruction tuning improves LLM performance beyond bilingual settings.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.