Streaming Speech-to-Text Translation with a SpeechLLM

2026-05-14 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new LLM-based architecture has been developed for real streaming speech-to-text translation, addressing the limitations of existing SpeechLLM systems that suffer from high latency. Traditional systems typically use separate modules for speech recognition and text-to-text translation, leading to cascaded errors and an inability to leverage paralinguistic information. This proposed SpeechLLM learns to determine when it has processed sufficient audio to emit output tokens, rather than waiting for a complete utterance or outputting at fixed intervals. The system is trained using automatic alignments between input speech and output text. Evaluations across various language pairs demonstrate that this architecture achieves translation quality comparable to non-streaming baselines, while significantly reducing latency to just 1-2 seconds.

Key takeaway

For research scientists developing real-time communication systems, this streaming SpeechLLM architecture offers a viable path to significantly reduce translation latency to 1-2 seconds without sacrificing quality. You should explore integrating dynamic token emission logic into your LLM-based translation models to enable more responsive and natural user experiences in live applications.

Key insights

A SpeechLLM can achieve real-time streaming speech-to-text translation with low latency and high quality.

Principles

• Combine speech recognition and translation.
• Exploit paralinguistic information.
• Reduce cascaded errors.

Method

An LLM learns to decide when to emit output tokens based on sufficient audio input, trained with automatic speech-text alignments.

In practice

• Integrate speech recognition and translation.
• Implement dynamic token emission.
• Utilize automatic alignment for training.

Topics

• Streaming Speech-to-Text
• SpeechLLM Architecture
• Low-Latency Translation
• Automatic Speech Alignment
• Large Language Models

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Streaming Speech-to-Text Translation with a SpeechLLM — A novel SpeechLLM adaptively decides when to emit translation tokens, achieving low latency and high quality.
• Low-Latency Real-Time Audio Game Commentary System via LLM-Based Parallel Text Generation — Parallel text generation and utterance buffering significantly reduce latency and improve rhythm in real-time audio commentary systems.
• How I Improved Speech-to-Text Accuracy — A two-pass LLM post-processing method effectively corrects common speech-to-text transcription errors.
• Voxtral Realtime — Voxtral Realtime delivers offline-quality ASR with sub-second latency via a natively streaming, end-to-end trained architecture.
• NaturalFlow: Reducing Disruptive Pauses for Natural Speech Flow in Simultaneous Speech-to-Speech Translation — NaturalFlow optimizes simultaneous speech-to-speech translation for natural flow by minimizing inter-chunk silences using internal model signals.
• Introducing Post-Stream Refinement: Higher-Accuracy Real-Time Transcription — Post-Stream Refinement enhances real-time speech recognition by combining instant partial results with a higher-accuracy, context-aware second pass.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.