How to Leverage Synthetic Speech for LLM-Based ASR Systems?

2026-07-11 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

A study on LLM-based Automatic Speech Recognition (ASR) systems addresses the challenge of training in regulated domains like banking and healthcare, where real speech data is costly and privacy-sensitive. Researchers investigated using synthetic speech from modern text-to-speech (TTS) to mitigate this. They probed a SLAM-ASR architecture, localizing the distributional gap between synthetic and real speech in the early-to-middle layers (0-14) of the LLM backbone, where temporal and prosodic perturbations are most disruptive. The work found that convolving synthetic audio with Room Impulse Responses (RIRs) effectively narrows this gap by introducing acoustic irregularities, not by improving perceptual quality. By combining a layer-selection module with RIR augmentation, the system matched a fully real-data baseline (8.68% WER) using only 25% of real speech (13.6 hours) and surpassed it at higher proportions, achieving 8.01% WER.

Key takeaway

For ML engineers developing ASR in regulated domains, you can significantly reduce reliance on sensitive real speech data. By incorporating RIR-augmented synthetic audio and a layer-wise weighted pooling module, your team can match or exceed a 100% real-data baseline with as little as 25% real speech. Focus on introducing acoustic irregularities to synthetic data rather than just improving its perceptual quality to bridge the domain gap effectively.

Key insights

RIR-augmented synthetic speech, combined with layer-wise pooling, significantly reduces real data needs for LLM-based ASR.

Principles

• The synthetic/real speech gap is concentrated in early-to-middle LLM layers.
• RIR augmentation bridges the acoustic domain gap by adding irregularities, not by improving naturalness.
• Layer-wise weighted pooling (LWP) is most beneficial in low-resource, substitution-heavy settings.

Method

A layer-wise weighted pooling (LWP) module learns per-token, per-utterance softmax-weighted combinations of all LLM layers, combined with Room Impulse Response (RIR) augmentation for synthetic speech.

In practice

• Use RIR augmentation for synthetic speech to mimic real-world acoustic conditions.
• Employ layer-wise weighted pooling in LLM-based ASR to optimize synthetic data utilization.
• Substitute 10-30% of real ASR training data with RIR-augmented synthetic speech for optimal performance.

Topics

• Automatic Speech Recognition
• Synthetic Speech
• Large Language Models
• Room Impulse Response
• Data Augmentation
• SLAM-ASR

Code references

• huggingface/parler-tts
• resemble-ai/chatterbox

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• NAVER LABS Europe Submission to the Instruction-following 2026 Short Track — The system combines an improved speech projector and synthetic domain-specific data for superior instruction-following speech processing.
• Improving multichannel speech enhancement through accurate room-acoustic simulations — High-fidelity room-acoustic simulations significantly improve multichannel speech enhancement performance, reducing word error rates by up to 38 %.
• Analyzing Error Propagation in Korean Spoken QA with ASR-LLM Cascades — ASR-stage information loss, especially single-character errors in Korean, is the primary bottleneck for ASR-LLM cascade performance.
• When Synthetic Speech Is All You Have: Better Call GRPO — GRPO, a critic-free reinforcement learning method, significantly outperforms SFT for LLM-based ASR adaptation using synthetic speech.
• Content Anonymization for Privacy in Long-form Audio — Linguistic content in long-form audio is a privacy risk, requiring content anonymization beyond voice disguise.
• Combining Real and Synthetic Speech for ASR Adaptation in Brazilian Portuguese — Domain-guided synthetic speech effectively augments real data for ASR adaptation in specialized, low-resource scenarios.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.