When Synthetic Speech Is All You Have: Better Call GRPO

2026-07-11 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Automatic Speech Recognition · Depth: Expert, extended

Summary

Group Relative Policy Optimization (GRPO) significantly improves LLM-based Automatic Speech Recognition (ASR) performance when adapting models using only synthetic text-to-speech (TTS) data, a common scenario in privacy-regulated domains like banking. This reinforcement learning approach reduces Word Error Rate (WER) by 40% relative to Supervised Fine-Tuning (SFT) on synthetic speech alone, dropping from 36.71% to 22.09%. An SFT-then-GRPO combination further improves this to 20.21%, a 45% relative reduction. The gain stems from GRPO's ability to reduce insertion errors by improving stopping calibration and anchoring attention to audio, rather than altering early-layer representations. GRPO also offers consistent, albeit smaller, gains on real data (10.27% to 9.49%).

Key takeaway

For Machine Learning Engineers adapting LLM-based ASR systems in privacy-sensitive domains with limited real speech, you should prioritize Group Relative Policy Optimization (GRPO) over Supervised Fine-Tuning (SFT) when using synthetic data. Your models will achieve significantly lower Word Error Rates, especially by focusing on WER as the reward function and strategically combining 5-10 hours of real speech with a larger, randomly sampled synthetic dataset. This approach prevents hallucinated insertions and improves attention anchoring.

Key insights

GRPO, a critic-free reinforcement learning method, significantly outperforms SFT for LLM-based ASR adaptation using synthetic speech.

Principles

• Reinforcement learning should be preferred over SFT for synthetic speech ASR adaptation.
• Optimizing the target metric directly (e.g., WER) yields the best reward function.
• Marginal value of real speech diminishes after 5-10 hours when combined with synthetic data.

Method

GRPO optimizes a sequence-level objective by sampling groups of hypotheses, standardizing rewards based on task metrics (e.g., 1-WER), and reinforcing better-performing outputs while suppressing lower-reward alternatives.

In practice

• Use GRPO for ASR domain adaptation in privacy-constrained settings.
• Prioritize WER as the reward function for GRPO training.
• Mix 5-10 hours of real speech with a larger synthetic pool for optimal WER reduction.

Topics

• Automatic Speech Recognition
• Reinforcement Learning
• Group Relative Policy Optimization
• Synthetic Speech
• LLM-based ASR
• Supervised Fine-Tuning
• Word Error Rate

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Diarization-Guided Qwen-ASR Adaptation for Multilingual Two-Speaker Conversational Speech — A diarization-guided, multi-stage ASR adaptation system significantly improves multilingual two-speaker conversational speech recognition.
• How to Leverage Synthetic Speech for LLM-Based ASR Systems? — RIR-augmented synthetic speech, combined with layer-wise pooling, significantly reduces real data needs for LLM-based ASR.
• AP-GRPO: Anchor-Gated Phonetic Alignment with Policy Optimization for Pathological Speech Reconstruction — AP-GRPO reconstructs pathological speech by aligning SLMs using audible anchors and phonetic compatibility, adapting to disease severity.
• Combining Real and Synthetic Speech for ASR Adaptation in Brazilian Portuguese — Domain-guided synthetic speech effectively augments real data for ASR adaptation in specialized, low-resource scenarios.
• Reinforcement Learning for LLM-based Event Forecasting — GRPO significantly enhances smaller LLMs' event forecasting beyond their knowledge cutoff.
• NAVER LABS Europe Submission to the Instruction-following 2026 Short Track — The system combines an improved speech projector and synthetic domain-specific data for superior instruction-following speech processing.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.