Mixed-Policy GRPO for Text-to-SQL with Off-Policy Data Generation

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, medium

Summary

Mixed-Policy Group Relative Policy Optimization (GRPO) for Text-to-SQL, presented by Sterbentz et al. at SURGeLLM 2026, addresses the inherent on-policy limitations of traditional GRPO in incorporating novel reasoning patterns. This new approach generates high-quality off-policy rollouts from existing datasets, facilitating mixed-policy training that exposes models to a wider array of reasoning trajectories. The research marks the first application of mixed-policy GRPO to the text-to-SQL domain and includes a systematic study of off-policy data generation, introducing Iterative Error Correction (IEC) for refining model outputs through targeted feedback. Experimental results demonstrate significant performance improvements, with mixed-policy GRPO outperforming base models by an average of +4.7% and on-policy GRPO by +4.1% across the Spider and BIRD benchmarks. Notably, gains on the BIRD benchmark reached up to +7.3% over base models and +4.5% over on-policy GRPO.

Key takeaway

For Machine Learning Engineers developing text-to-SQL solutions, consider adopting mixed-policy GRPO to overcome limitations of on-policy training. Your models can achieve substantial performance gains, averaging +4.7% over base models, by incorporating diverse off-policy reasoning trajectories. Implement Iterative Error Correction to systematically refine model outputs, particularly for complex benchmarks like BIRD where gains reached +7.3%. This approach directly enhances your model's ability to handle novel and complex SQL generation tasks.

Key insights

Mixed-policy GRPO with off-policy data generation significantly enhances text-to-SQL reasoning by diversifying training trajectories.

Principles

• On-policy methods limit novel reasoning.
• Diverse training data improves model performance.
• Iterative feedback refines model outputs.

Method

Mixed-policy GRPO trains text-to-SQL models using off-policy rollouts generated from existing datasets. It incorporates Iterative Error Correction (IEC) to refine model outputs through targeted feedback, exposing models to varied reasoning trajectories.

In practice

• Apply mixed-policy GRPO to text-to-SQL tasks.
• Generate off-policy data from existing datasets.
• Implement Iterative Error Correction for refinement.

Topics

• Text-to-SQL
• Group Relative Policy Optimization
• Off-Policy Learning
• Iterative Error Correction
• Natural Language Processing
• Reinforcement Learning

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Selective-Advantage Entropy-Adaptive Horizon GRPO: Asymmetric Token-Level Discounting for Efficient Reinforcement Learning of Language Models — Asymmetric token-level discounting in GRPO significantly stabilizes and improves reinforcement learning for language models.
• SIRIUS-SQL: Anchoring Multi-Candidate Text-to-SQL in Execution Feedback — Multi-candidate Text-to-SQL reliability improves by diversifying generation, targeting repairs, and hybrid selection.
• On-policy distillation: one of the hottest terms on PapersWithCode [R] — On-policy distillation corrects specific model errors by using a hint model to guide probability adjustments without new rollouts.
• Improving Cross-Lingual Factual Recall via Consistency-Driven Reinforcement Learning — GRPO significantly improves cross-lingual factual recall and consistency in LLMs by fostering shared representations, outperforming SFT.
• CATPO: Critique-Augmented Tree Policy Optimization — CATPO optimizes LLM reasoning by focusing RL training on informative trees and healing failures with critiques.
• Semi-Offline Reinforcement Learning for Optimized Text Generation — The semi-offline RL paradigm optimizes text generation by blending online exploration and offline efficiency using masked observations.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.