Are you speaking my languages? On spoken language adherence in multimodal LLMs

2026-06-15 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Multimodal Large Language Models (LLMs) used for Automatic Speech Recognition (ASR) frequently misidentify the output language, which degrades transcription accuracy and downstream application quality, despite enabling seamless multilingual use. Researchers propose a soft prompting approach to address this by hinting at potential spoken languages without strict constraints, preserving flexibility and code-switching. They formally define this issue as a lack of language adherence, introduce a novel metric to quantify violations, and evaluate three mitigation strategies: zero-shot prompting, supervised fine-tuning (SFT), and Chain-of-Thought (CoT) reasoning. A comparative analysis across multiple languages assesses their effectiveness in reducing language violations while maintaining overall ASR performance, discussing trade-offs for strategy selection under various compute constraints.

Key takeaway

For Machine Learning Engineers developing multilingual ASR systems with LLMs, addressing language adherence is crucial for transcription quality. You should consider implementing soft prompting to guide output language without sacrificing code-switching flexibility. Evaluate zero-shot prompting for robust guidance, supervised fine-tuning for improved adherence, or Chain-of-Thought reasoning for enforcement during decoding, carefully weighing each strategy's effectiveness against your specific compute constraints to optimize performance.

Key insights

Multimodal LLMs require soft prompting and specific strategies to ensure spoken language adherence in ASR, preventing transcription errors.

Principles

• Language adherence is critical for LLM-based ASR fidelity.
• Soft prompting guides language without strict constraints.
• Zero-shot, SFT, and CoT improve prompt adherence.

Method

The research defines language adherence, introduces a novel violation metric, and evaluates zero-shot prompting, supervised fine-tuning, and Chain-of-Thought reasoning to mitigate output language misidentification in ASR.

In practice

• Implement soft prompting for multilingual ASR.
• Evaluate zero-shot, SFT, or CoT for language adherence.
• Select mitigation strategies based on compute constraints.

Topics

• Multimodal LLMs
• Automatic Speech Recognition
• Language Adherence
• Soft Prompting
• Supervised Fine-tuning
• Chain-of-Thought Reasoning
• Multilingual ASR

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Can AI Save India’s Dying Languages? | Building a Tulu LLM Without Training Data Ft. Prathamesh — Structured prompting and custom romanization enable LLMs to generate low-resource languages without extensive training data.
• Evaluating Large Language Models Abilities for Addressee, Turn-change, and Next Speaker Prediction in Meetings — Text-based LLMs excel at next speaker prediction in meetings, even without multimodal input.
• LANG: Reinforcement Learning for Multilingual Reasoning with Language-Adaptive Hint Guidance — LANG enhances multilingual LLM reasoning by using language-adaptive hints with a progressive decay schedule, preventing language drift.
• References Improve LLM Alignment in Non-Verifiable Domains — Reference-guided LLM evaluators significantly improve model alignment and self-improvement in non-verifiable domains.
• Can LLMs Be Constrained to the Past? Improving Knowledge Cutoff through Recall-Based Prompting — LLMs can better adhere to knowledge cutoffs using recall-based prompting, especially for causally related information.
• IndicContextEval: A Benchmark for Evaluating Context Utilisation in Audio Large Language Models Across 8 Indic Languages — AudioLLMs vary significantly in how they utilize contextual prompts, with native-script entities and natural language being most effective.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.