What Predicts Correctness in Text-to-SQL? A Selective-Prediction Study

2026-07-07 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

A study on predicting correctness in AI-generated SQL queries, specifically for hard multi-table text-to-SQL tasks, evaluates various signals using AUROC. On BIRD and Spider benchmarks, black-box signals like string, structural, and execution self-consistency, schema-relevance, and query executability achieve AUROC scores between 0.61 and 0.68, with string self-consistency at 0.675. White-box log-probability is similar at 0.67. Verification-based signals, particularly LLM judges, significantly outperform these, with GPT-4o-mini scoring 0.72 and Claude reaching 0.78 AUROC. An ensemble of two different LLM judge providers achieves 0.82 AUROC with a well-calibrated probability (0.03 ECE), enabling useful abstention frontiers, such as answering 27% of questions at 24% selective risk. While fine-tuned verifiers show strong in-distribution performance (0.77-0.79 AUROC), they struggle with unseen schemas (0.66 AUROC), a gap that scaling, schema diversity, or distillation fails to close. Generalization across schemas currently requires large, frozen reasoning models.

Key takeaway

For Machine Learning Engineers deploying Text-to-SQL solutions, you should integrate reasoning-based LLM verifiers to significantly improve query correctness prediction. An ensemble of judges from different providers, like Claude and GPT-4o-mini, achieves 0.82 AUROC, enabling you to confidently abstain from 27% of questions at 24% risk. This approach offers a robust method for managing uncertainty and enhancing system reliability.

Key insights

Verification-based LLM judges significantly improve Text-to-SQL correctness prediction over black-box methods, especially when ensembled.

Principles

• Verification-based signals outperform black-box and white-box methods.
• Ensembling multiple LLM judges enhances prediction accuracy and calibration.
• Cross-schema generalization for verifiers requires large reasoning models.

Method

Combine LLM judges from different providers to create an ensemble verifier, achieving higher AUROC and better calibration for Text-to-SQL correctness prediction.

In practice

• Use LLM judges (e.g., Claude, GPT-4o-mini) for Text-to-SQL verification.
• Implement a two-provider LLM judge ensemble for robust correctness prediction.
• Employ fine-tuned verifiers for in-domain Text-to-SQL correctness tasks.

Topics

• Text-to-SQL
• Query Correctness Prediction
• LLM Verification
• Selective Prediction
• AUROC
• Model Generalization

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Scaling LLM Reasoning from Minimal Labels: A Semi-Supervised Framework with a Lightweight Verifier — A semi-supervised framework scales LLM reasoning by using a lightweight verifier to generate high-confidence pseudo-labels from minimal human input.
• Adaptive Conformal Prediction for Improving Factuality of Generations by Large Language Models — Adaptive conformal prediction improves LLM factuality by calibrating uncertainty estimates based on specific prompts.
• Mamba-SSM with LLM Reasoning for Feature Selection: Faithfulness-Aware Biomarker Discovery — LLM reasoning significantly improves biomarker discovery performance by precisely removing confounders, even with imperfect biological recall.
• Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs — LLM uncertainty estimation in multilingual settings is significantly impacted by reasoning language and model scale.
• Understanding, Detecting, and Repairing Real-World In-Context-Learning-Based Text-to-SQL Errors — LLM-generated SQL queries have widespread, categorized errors, requiring targeted, efficient repair solutions.
• When the Judge Is Wrong — LLM-as-judge is unreliable for structured tasks, even with ground truth or source documents.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.