We Need SQL for AI Programs (And IBM Might Have Built It)

2026-01-06 · Source: Towards AI - Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, quick

Summary

IBM Research has introduced "Span Queries," a novel approach designed to optimize Generative AI programs by applying principles similar to database query optimization. The current challenge in building with Large Language Models (LLMs) is the lack of tools for optimizing chained API calls, leading to inefficient execution and unpredictable performance. Span Queries aim to address this by allowing developers to declaratively describe GenAI programs, much like SQL queries describe database operations. This enables an underlying optimizer to determine the most efficient execution path, potentially reordering operations, utilizing caching, or parallelizing tasks, thereby moving beyond the current "flying blind" method of LLM program development.

Key takeaway

For AI Architects and Research Scientists building complex LLM applications, Span Queries offer a potential solution to the current lack of optimization tools. Your teams can move from imperative, trial-and-error LLM chaining to a more declarative, SQL-like approach, allowing an underlying optimizer to handle execution efficiency. This could significantly improve performance and reliability, reducing the need for manual tuning and guesswork in production systems.

Key insights

Span Queries propose a declarative optimization layer for GenAI programs, akin to SQL for databases.

Principles

• Declarative programming enables optimization.
• Optimizers improve efficiency without manual tuning.

Method

Describe GenAI programs declaratively, allowing an optimizer to determine efficient execution paths, including reordering, caching, and parallelization, similar to how database engines optimize SQL queries.

In practice

• Define GenAI workflows declaratively.
• Leverage automatic optimization for LLM chains.

Topics

• LLM Optimization
• Span Queries
• Declarative AI Programming
• GenAI Programs

Best for: AI Architect, AI Scientist, Research Scientist, Machine Learning Engineer, AI Engineer, AI Researcher

Related on AIssential

• Multi-Agent Architecture with RAG and Dynamic Context Windows for Text-to-SQL Optimization — A multi-agent Text-to-SQL system uses RAG and dynamic context to reduce token consumption.
• QueryWeaver: Reliable Multi-Tool Query Execution Planning via LLM-Based Graph Generation — QueryWeaver transforms natural language into structured graphs for reliable multi-tool query execution using a deterministic, dependency-resolving planner.
• Benchmarking Local LLMs for Natural-Language-to-SQL Querying in Biopharmaceutical Manufacturing: An Empirical Benchmark on Consumer-Grade Hardware — Code-tuned general-purpose LLMs outperform domain-specific medical LLMs for T-SQL generation on manufacturing schemas.
• When Gradients Collide: Failure Modes of Multi-Objective Prompt Optimization for LLM Judges — Multi-objective prompt optimization for LLM judges fails due to gradient dilution and instruction interference.
• TAHOE: Text-to-SQL with Automated Hint Optimization from Experience — TAHOE optimizes Text-to-SQL LLM prompts by dynamically learning and applying error-driven hints from development and deployment experiences.
• TopBench: A Benchmark for Implicit Predictive Reasoning in Tabular Question Answering — LLMs struggle with implicit predictive reasoning in tabular data, often misinterpreting intent as simple retrieval.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards AI - Medium.