Unlocking Speech-Text Compositional Powers: Instruction-Following Speech Language Models without Instruction Tuning

2026-07-02 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

SpeechCombine is a novel instruction-following speech language model (SLM) designed to overcome the challenges of instruction tuning, which is significantly more complex for speech than for text-based large language models (LLMs). Unlike current approaches that synthesize extensive speech pre-training and instruction-tuning datasets, a strategy difficult to scale due to longer speech sequences, SpeechCombine operates without any instruction tuning. It achieves this by utilizing a single round of speech pre-training on 30k hours of data. The model begins with a text LLM base, undergoes continuous pre-training on speech utterances to adapt, and then directly combines its weights with the weight difference between the instruction-tuned and base versions of the original text LLM. This simple combination effectively transfers the text LLM's knowledge and capabilities to the speech domain, proposing a new SLM training paradigm that reduces reliance on massive speech data.

Key takeaway

For Machine Learning Engineers developing new speech language models, SpeechCombine offers a critical alternative to traditional instruction tuning. You can now adapt existing text LLMs for speech tasks by combining weights, significantly reducing the need for extensive speech-specific instruction datasets. This approach streamlines SLM development, utilizing pre-trained text knowledge more efficiently and accelerating deployment with less data overhead.

Key insights

SpeechCombine enables instruction-following SLMs without instruction tuning by weight combination, reducing reliance on massive speech data.

Principles

• Instruction tuning for SLMs is difficult to scale.
• Weight combination can transfer LLM knowledge to speech.
• Avoids massive speech data reliance.

Method

Start with a text LLM base, continuously pre-train on speech, then combine its weights with the instruction-tuned/base text LLM weight difference.

In practice

• Develop SLMs without extensive instruction tuning.
• Adapt existing text LLMs for speech tasks.
• Reduce speech data collection needs.

Topics

• Speech Language Models
• Instruction Tuning
• Large Language Models
• Model Weight Combination
• Speech Pre-training
• Multimodal AI

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Large Language Models in NLP — LLMs integrate NLP advancements with massive scale to create general-purpose, prompt-driven language systems.
• LEAF: Growing Trees Without Branching for Speech-Aware Large Language Model Post-Training — LEAF is a retrospective tree-based RL method for speech-aware LLM post-training that refines credit assignment.
• Short-form Text Rewriting with Phi Silica — Targeted adaptation of Small Language Models (SLMs) can significantly improve their performance for precision-critical short-form text rewriting tasks.
• Reinforced Fast Weights with Next-Sequence Prediction — REFINE improves fast weight models for long-context tasks by shifting from next-token to next-sequence prediction via reinforcement learning.
• A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAM$Δ$ Integration into Upcycled MoE — Upcycling dense LLMs into MoE architectures with parameter delta integration efficiently expands language capabilities.
• LLMs on Tabular Data with Limited Semantics: Evidence from Industrial Car Retrofit Prediction — LLMs complement, but do not replace, strong tabular models on privacy-constrained industrial data lacking semantic cues.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.