Spider 2.0-AIFunc: Extending Real-World Text-to-SQL to AI-Native SQL Workflows

2026-07-07 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

Spider 2.0-AIFunc is a new benchmark designed to evaluate text-to-SQL models on AI-native SQL workflows, addressing a gap where existing benchmarks only cover conventional SQL. This benchmark comprises 465 verified instances across 125 real-world databases, specifically focusing on six types of AI functions available on the Snowflake platform, such as classification and sentiment analysis. It was constructed using an agent-based pipeline that rewrites source tasks into AI-native forms, simultaneously transforming target queries and refining natural language instructions to reduce ambiguity. Evaluation of ten state-of-the-art language models revealed that top proprietary models achieved 67-70% execution accuracy, while the best open-source model reached 58.1%. The performance gap is primarily due to errors in predicate specification, schema grounding, and AI function parameterization. Interestingly, traditional text-to-SQL agent frameworks did not effectively transfer, with minimal agent setups often outperforming more complex alternatives.

Key takeaway

For Machine Learning Engineers developing text-to-SQL solutions for modern data platforms, you should recognize that AI-native SQL functions introduce new complexities not covered by traditional benchmarks. Focus your model development on improving predicate specification, schema grounding, and AI function parameterization, as these are critical error sources. Consider evaluating your models against the Spider 2.0-AIFunc benchmark to accurately assess their performance on real-world AI-native SQL workflows, and don't assume complex agent frameworks will automatically yield better results.

Key insights

AI-native SQL workflows demand specialized benchmarks, revealing current LLM limitations in predicate and AI function parameterization.

Principles

• AI-native SQL presents distinct challenges for text-to-SQL models.
• Proprietary models significantly outperform open-source on AI-native SQL.
• Simpler agent setups can be more effective for AI-native SQL.

Method

An agent-based pipeline rewrites source tasks into AI-native forms, transforms target queries, and refines natural language instructions. Instances undergo multi-round execution for stability.

In practice

• Prioritize improving predicate specification and AI function parameterization in models.
• Test minimal agent frameworks for AI-native text-to-SQL tasks.
• Benchmark models using Spider 2.0-AIFunc for AI-native SQL evaluation.

Topics

• Text-to-SQL
• AI-native SQL
• LLM Benchmarking
• Snowflake Platform
• Agent Frameworks
• Database Querying

Code references

• Leolty/Spider2-AIFunc

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• How I Replaced 1,000 Brittle Rules with 3 AI Calls: A Hybrid Data Quality Framework — Combine deterministic SQL rules with inline AI for scalable, context-aware data quality, reducing false positives and cost.
• From Natural Language to PromQL: A Catalog-Driven Framework with Dynamic Temporal Resolution for Cloud-Native Observability — A catalog-driven framework translates natural language into PromQL, simplifying cloud-native observability with dynamic temporal resolution.
• The TechBeat: The IDE Is No Longer Just a Place to Write Code (6/17/2026) — AI is deeply integrating into development, operations, and business strategies, demanding new approaches.
• AI Builders: Building a text-to-SQL agent — Text-to-SQL agents democratize data access by translating natural language queries into structured database commands.
• BADGER: Bridging Agentic and Deterministic Evaluation for Generative Enterprise Reasoning — BADGER unifies text-to-SQL and agentic reasoning evaluation for enterprise AI, validated by human expert judgment.
• How I Added AI-Powered Natural Language Parsing to My iOS App with Firebase Cloud Functions — Domain-specific context and server-side LLM calls significantly enhance natural language parsing for consumer apps.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.