Spider 2.0-AIFunc: Extending Real-World Text-to-SQL to AI-Native SQL Workflows

2026-07-08 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Software Development & Engineering · Depth: Expert, extended

Summary

Spider 2.0-AIFunc is a new benchmark for evaluating text-to-SQL systems on AI-native SQL workflows, specifically incorporating large language model capabilities exposed as native SQL functions on platforms like Snowflake. It comprises 465 verified instances across 125 real-world databases, covering six types of Snowflake Cortex AI Functions: AI_CLASSIFY, AI_FILTER, AI_SENTIMENT, AI_SIMILARITY, AI_EXTRACT, and AI_AGG. Constructed via an agent-based pipeline that rewrites existing enterprise text-to-SQL tasks, the benchmark ensures result stability through a multi-round execution verification protocol. Evaluation of ten state-of-the-art language models revealed proprietary models (e.g., Claude Opus 4.6 at 70.3%) significantly outperform open-source models (best at 58.1%), with errors primarily in predicate specification, schema grounding, and AI function parameterization. Traditional text-to-SQL agent frameworks did not effectively transfer to this AI-native context.

Key takeaway

For Machine Learning Engineers or Data Scientists building AI-native SQL applications, you should prioritize proprietary LLMs for text-to-SQL tasks involving AI functions, given their superior execution accuracy (67-70% vs. 58.1% for open-source). Be prepared to meticulously refine schema grounding, predicate specification, and AI function parameterization, as these are common error sources. Consider direct integration of AI functions into SQL rather than relying on traditional text-to-SQL agent frameworks, which proved less effective in this context.

Key insights

The benchmark reveals a significant performance gap between proprietary and open-source LLMs in generating AI-native SQL.

Principles

• AI-native SQL requires precise instruction and parameter specification for deterministic evaluation.
• Traditional text-to-SQL agent frameworks are less effective for AI-native SQL challenges.
• LLM outputs can exhibit minor inconsistencies even at temperature zero, necessitating verification.

Method

The benchmark was constructed using an "agent-based pipeline" that rewrites source tasks into AI-native form, transforms target queries, refines natural language instructions, and employs multi-round execution verification for stability.

In practice

• Integrate AI functions like AI_SENTIMENT and AI_CLASSIFY directly into SQL queries for semantic operations.
• Use Snowflake Cortex AI Functions for classification, filtering, sentiment analysis, similarity search, extraction, and aggregation.

Topics

• Text-to-SQL
• AI-Native SQL
• Snowflake Cortex AI Functions
• LLM Benchmarking
• Agent Frameworks
• SQL Generation

Code references

• Leolty/Spider2-AIFunc

Best for: Research Scientist, AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer, Data Scientist

Related on AIssential

• Understanding, Detecting, and Repairing Real-World In-Context-Learning-Based Text-to-SQL Errors — LLM-generated SQL queries have widespread, categorized errors, requiring targeted, efficient repair solutions.
• AgentNLQ: A General-Purpose Agent for Natural Language to SQL — Multi-agent orchestration with enriched schema and execution feedback significantly improves NL2SQL accuracy and efficiency.
• Production-Grade Text-to-SQL Agent with Claude Code, LangGraph, Langfuse, FastAPI and Qdrant — Building robust Text-to-SQL agents requires RAG, explicit state machines, and rigorous evaluation to prevent hallucinations and ensure reliability.
• Data Machina #262 — New models and frameworks are advancing LLM efficiency, reasoning, and long-form content generation.
• Benchmarking Local LLMs for Natural-Language-to-SQL Querying in Biopharmaceutical Manufacturing: An Empirical Benchmark on Consumer-Grade Hardware — Code-tuned general-purpose LLMs outperform domain-specific medical LLMs for T-SQL generation on manufacturing schemas.
• How I Added AI-Powered Natural Language Parsing to My iOS App with Firebase Cloud Functions — Domain-specific context and server-side LLM calls significantly enhance natural language parsing for consumer apps.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.