Rethinking Speech-LLM Integration for ASR: Effective Joint Speech-Text Training by Interleaving

2026-07-02 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Speech and Language Processing · Depth: Expert, quick

Summary

Joint Speech-Text Interleaved Pretraining (JSTIP) is a novel ASR-oriented pretraining strategy designed to enhance automatic speech recognition by effectively integrating speech-LLM architectures. Addressing the observation that LLM priors become less evident with increasing supervised ASR data, JSTIP constructs word-level and segment-level interleaved speech-text sequences within aligned pairs. Experiments conducted on 38k hours of ASR data demonstrate consistent entity accuracy improvements over ASR-only and standard joint speech-text training baselines. JSTIP also achieves comparable entity recognition performance using readily available domain transcription text, simplifying domain adaptation compared to relying on synthetic speech-text pairs. Published on 2026-07-02, this method is competitive with existing open-source ASR and Speech-LLM systems in medical entity recognition, suggesting it reduces the speech-text modality gap and preserves LLM generative priors.

Key takeaway

For Machine Learning Engineers developing ASR systems, especially those struggling with entity recognition or domain adaptation, consider implementing Joint Speech-Text Interleaved Pretraining (JSTIP). This approach allows you to leverage textual knowledge more effectively, improving entity accuracy on your ASR data. You can simplify domain adaptation by utilizing existing domain transcription text instead of costly synthetic speech-text pairs, making your development process more efficient and robust.

Key insights

JSTIP effectively integrates speech and text for ASR by interleaving sequences, improving entity recognition and reducing modality gaps.

Principles

• LLM priors' ASR contribution diminishes with more supervised data.
• Interleaving speech-text sequences reduces modality gaps.
• Domain transcription text can replace synthetic pairs for adaptation.

Method

JSTIP constructs word-level and segment-level interleaved speech-text sequences within aligned pairs for speech-LLM architectures that accept continuous inputs.

In practice

• Improve ASR entity accuracy using interleaved speech-text pretraining.
• Simplify domain adaptation with readily available domain transcription text.
• Enhance zero-shot speech question answering behaviors.

Topics

• Automatic Speech Recognition
• Speech-LLM Integration
• Joint Speech-Text Training
• Interleaved Pretraining
• Entity Recognition
• Domain Adaptation

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR — NIM4-ASR optimizes LLM-based ASR for efficiency and robustness by redesigning multi-stage training and integrating production-ready features.
• A Comparative Study of Pretrained Transformer Models for Quranic ASR: Speech Representations, Label Formats, and Dataset Composition — Fine-tuning pretrained Transformers with specific speech representations and label formats significantly improves Quranic ASR accuracy and efficiency.
• Towards Truly Multilingual ASR: Generalizing Code-Switching ASR to Unseen Language Pairs — Code-switching ASR generalization to unseen language pairs remains limited despite model merging and domain generalization efforts.
• Are you speaking my languages? On spoken language adherence in multimodal LLMs — Multimodal LLMs require soft prompting and specific strategies to ensure spoken language adherence in ASR, preventing transcription errors.
• Ontology Memory-Augmented ASR Correction for Long Text-Speech Interleaved Conversations — An ontology memory framework enhances ASR correction by structuring conversation history for context-grounded evidence in long, interleaved dialogues.
• Combining Real and Synthetic Speech for ASR Adaptation in Brazilian Portuguese — Domain-guided synthetic speech effectively augments real data for ASR adaptation in specialized, low-resource scenarios.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.