Simultaneous Speech-to-Speech Translation Without Aligned Data

2025-04-30 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, extended

Summary

Hibiki-Zero is a novel simultaneous speech-to-speech (S2ST) and speech-to-text (S2TT) translation system that eliminates the need for word-level aligned training data. It achieves this by initially training on sentence-level aligned data to learn high-latency speech translation, followed by a reinforcement learning (RL) strategy using GRPO to optimize latency while preserving translation quality. The model, with 3 billion parameters, leverages a multistream architecture and a simple BLEU score-based reward system for RL. Hibiki-Zero demonstrates state-of-the-art performance in translation accuracy, latency, voice transfer, and naturalness across five X-to-English tasks, outperforming baselines like Seamless and Hibiki. It also shows strong adaptability, supporting new input languages with less than 1000 hours of speech data. Model weights, inference code, and a 45-hour multilingual benchmark are publicly released.

Key takeaway

For Machine Learning Engineers developing real-time speech translation systems, Hibiki-Zero offers a compelling approach to overcome data alignment challenges. You should consider adopting its two-stage training, starting with coarse sentence-level alignments and then applying reinforcement learning with BLEU-based process rewards to fine-tune latency and quality. This method significantly simplifies data preparation and enables rapid adaptation to new languages with minimal data, potentially accelerating multilingual S2ST deployment.

Key insights

Simultaneous S2ST/S2TT can achieve state-of-the-art performance without word-level aligned data, using RL to optimize latency.

Principles

• Sentence-level alignments simplify S2ST training.
• RL with process rewards optimizes latency and quality.
• Multistream architecture supports real-time S2ST.

Method

Hibiki-Zero trains a base model on sentence-level aligned data, then refines its translation policy using GRPO-based reinforcement learning with process rewards derived from BLEU scores at intermediate translation instants.

In practice

• Adapt S2ST models to new languages with <1000h data.
• Utilize GRPO with BLEU process rewards for latency optimization.
• Explore multistream architectures for real-time S2ST.

Topics

• Simultaneous Speech Translation
• Speech-to-Speech Translation (S2ST)
• Reinforcement Learning
• Multilingual NLP
• Low-Resource Language Adaptation
• Audio Codecs

Code references

• kyutai-labs/hibiki-zero
• openai/whisper
• Unbabel/COMET
• microsoft/UniSpeech

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• A Pocket Offline Model for Simultaneous Speech Translation as CUNI Submission to IWSLT 2026 — A 1-billion-parameter model enables simultaneous speech translation on pocket devices using Nemo toolkit and AlignAtt policy.
• VTaMo: Video-Text Alignment Model for Sign Language Translation — Explicit multi-granularity alignment significantly improves gloss-free sign language translation by resolving vision-text semantic and temporal mismatches.
• Kyutai Releases Hibiki-Zero: A3B Parameter Simultaneous Speech-to-Speech Translation Model Using GRPO Reinforcement Learning Without Any Word-Level Aligned Data — Hibiki-Zero enables simultaneous speech translation without word-level alignments using GRPO reinforcement learning.
• CUHKSZ Simultaneous Speech Translation System for IWSLT 2026 — A simultaneous speech translation system internalizes read/write policies for low-latency, high-BLEU performance using an audio-text LLM.
• NaturalFlow: Reducing Disruptive Pauses for Natural Speech Flow in Simultaneous Speech-to-Speech Translation — Simultaneous S2ST can achieve natural speech flow by optimizing for fluency and translation quality concurrently, avoiding disruptive pauses.
• VoiceTTA: Enhancing Zero-Shot Text-to-Speech via Reinforcement Learning-Based Test-Time Adaptation — VoiceTTA enhances zero-shot TTS for uncommon styles via RL-based test-time adaptation, optimizing learnable prefixes with novel F0/energy style rewards.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.