Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervision

2026-07-27 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, long

Summary

A novel language-aware distillation framework addresses language interference in multilingual Speech Large Language Models (LLMs) that follow instructions. This method, which uses a query bank and a gating network with a Q-Former projector, dynamically selects or mixes query tokens to disentangle language-specific information. Unlike prior approaches requiring extensive supervised fine-tuning, this framework trains performant multilingual Speech LLMs efficiently with only 5.8K hours of ASR data across six languages, keeping the Whisper-large-v3 speech encoder and Llama-SEA-LION-v3-8B-IT LLM frozen. The approach demonstrates a 14% gain over matched multilingual distillation baselines on instruction following and improves over existing Speech LLM baselines by 32% on the newly synthesized Audio–MLQA benchmark. Ablation studies confirm hard query selection's superiority and high language identification accuracy.

Key takeaway

For Machine Learning Engineers developing multilingual speech understanding systems, this language-aware distillation framework offers a resource-efficient paradigm. You should consider implementing dynamic query banks with hard selection to mitigate language interference, especially for lower-represented languages. This approach allows you to achieve strong performance across diverse languages using only ASR data, preserving your frozen LLM and speech encoder from catastrophic forgetting.

Key insights

Language-aware distillation with dynamic query selection resolves multilingual interference in Speech LLMs using ASR-only data.

Principles

• Dynamic query selection improves multilingual Speech LLM performance.
• Hard query selection decouples language-specific information better.
• Increased query length enhances phonetic-semantic mapping.

Method

A query bank and gating network (convolutional LID or attention-pooling MLP) select or mix query tokens for a Q-Former projector, feeding a frozen LLM. Training uses LID, input, and output distillation losses.

In practice

• Implement a query bank with hard selection for multilingual speech tasks.
• Freeze backbone LLM and speech encoder to prevent catastrophic forgetting.
• Synthesize high-quality TTS data for multilingual QA benchmarks.

Topics

• Speech LLMs
• Multilingual Understanding
• Context Distillation
• Q-Former
• ASR-only Supervision
• Audio-MLQA

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Multilingual Chain-of-Thought Compression via Cross-Lingual Distillation — MCD compresses multilingual Chain-of-Thought reasoning using cross-lingual data alignment and reinforcement learning, improving efficiency and low-resource language robustness.
• NeMo@IWSLT 2026: Cascaded System for Simultaneous Speech Translation — A cascaded ASR-LLM system achieves stable, high-quality simultaneous speech translation across diverse languages and latencies.
• Same Evidence, Different Answers: Canonical-Context On-Policy Distillation for Multi-Turn Language Models — Aligning multi-turn LLM behavior with full-context teacher responses mitigates self-anchored drift.
• Audio Sentiment Analysis via Distillation and Cross-Modal Integration of Generated Multilingual Transcripts — Multimodal integration of audio and generated multilingual text significantly boosts audio sentiment analysis performance.
• MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognition — MoLGE uses language grouping and LoRA to efficiently scale multilingual ASR, overcoming capacity dilution.
• PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects — PolySpeech-100 reveals E2E Speech-LLMs excel on dialects, but open-source models struggle with low-resource languages, and CoT prompting can degrade performance.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.