Do Factual Recall Mechanisms Carry over from Text to Speech in Multimodal Language Models?

2026-05-22 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, extended

Summary

A study investigated how factual recall mechanisms transfer from text to speech in multimodal Speech Language Models (SLMs), focusing on SpiritLM. Using Causal Mediation Analysis (CMA) on the Known dataset, researchers compared text-to-text (T->T) and speech-to-text (S->T) factual recall. Results showed that SpiritLM's text-processing pathways maintained strong causal signals for factual recall, consistent with text-only models, particularly in mid-layer MLPs. However, when speech input was used, the Average Indirect Effect (AIE) for factual recall significantly decreased, indicating a weaker, more diffuse signal, although still present around subject tokens in MLP and attention layers. This suggests that while factual associations in SLMs are not strictly modality-dependent, the mechanisms for recalling facts are only partially carried over from text to speech, with text serving as a more reliable trigger. The speech dataset was synthesized using MeloTTS, and transcription validation with Whisper-small showed a 19% Word Error Rate.

Key takeaway

For NLP Engineers developing multimodal language models, you should recognize that factual recall mechanisms may not fully transfer from text to speech. If you are fine-tuning SLMs on speech data, prioritize strategies that explicitly bridge the semantic gap between modalities to ensure robust knowledge retrieval. Evaluate your models using causal tracing in both text-to-text and speech-to-text settings to identify and address modality-specific weaknesses in factual association.

Key insights

Factual recall mechanisms in multimodal SLMs are weaker and less localized for speech inputs compared to text.

Principles

• Factual knowledge in LLMs localizes in mid-layer feed-forward networks.
• Multimodal SLMs may not fully transfer text-learned recall to speech.
• Semantic gaps between modalities can compromise knowledge transfer.

Method

Causal Mediation Analysis (CMA) with clean, corrupted, and corrupted-with-restoration runs quantifies component contributions to factual prediction. Extended with CTC-based forced alignment for speech inputs.

In practice

• Use CMA to pinpoint factual knowledge localization in multimodal models.
• Evaluate speech-enabled AI systems for factual recall discrepancies.
• Consider modality alignment in SLM fine-tuning for better transfer.

Topics

• Speech Language Models
• Factual Recall
• Causal Mediation Analysis
• Multimodal AI
• SpiritLM
• Mechanistic Interpretability
• Speech-to-Text

Code references

• myshell-ai/MeloTTS

Best for: Research Scientist, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• Do Factual Recall Mechanisms Carry over from Text to Speech in Multimodal Language Models? — Factual recall mechanisms in multimodal SLMs only partially transfer from text to speech.
• Revisiting Non-Verbatim Memorization in Large Language Models: The Role of Entity Surface Forms — LLM factual recall is sensitive to entity surface forms, highlighting the need for diverse evaluation methods.
• Disentangling Linguistic Relatedness from Task Alignment in Cross-Lingual Transfer — Cross-lingual fine-tuning primarily improves task-format alignment, not language-specific knowledge transfer, especially for weaker models.
• Understanding Cross-Modal Contributions in Continual Vision-Language Models: A Theoretical Perspective — A new theoretical perspective clarifies cross-modal contributions in continual vision-language models, addressing catastrophic forgetting.
• How Multimodal AI Actually Works: A Founder’s Plain-English Breakdown — Multimodal AI synthesizes diverse data streams to infer content intent, improving moderation accuracy beyond unimodal detection.
• Ontology Memory-Augmented ASR Correction for Long Text-Speech Interleaved Conversations — An ontology memory framework enhances ASR correction by structuring conversation history for context-grounded evidence in long, interleaved dialogues.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.