NAVER LABS Europe Submission to the Instruction-following 2026 Short Track

2026-07-02 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

NAVER LABS Europe submitted an advanced system to the IWSLT 2026 instruction-following speech processing short track, achieving a tie for first place. This system jointly performs Automatic Speech Recognition (ASR), Speech Translation (ST), and Speech Question Answering (SQA) from English speech into Chinese, Italian, and German within a constrained setting. Building on their previous top-ranked submission, the team updated their multi-stage training pipeline. Key improvements include replacing the speech projector with SpeechMapper, a novel method that learns a speech-to-LLM embedding projector using only ASR data. Additionally, they introduced fakACL, a synthetic SQA dataset comprising artificially generated scientific presentations. This dataset was created by prompting an LLM backbone, segmenting the generated talks, and synthesizing speech with SeamlessM4T-large-v2. The combined enhancements resulted in a more compact model that outperforms last year's best short-track system, despite relying on a weaker LLM backbone.

Key takeaway

For Machine Learning Engineers developing multi-task speech processing systems, this work highlights a path to superior performance and efficiency. You should explore generating domain-specific synthetic data using LLMs and advanced text-to-speech models like SeamlessM4T-large-v2 to augment your training sets. Additionally, consider SpeechMapper's approach of learning speech-to-LLM embeddings solely from ASR data, as it enables more compact models with weaker LLM backbones to achieve top-tier results in complex tasks like joint ASR, ST, and SQA.

Key insights

The system combines an improved speech projector and synthetic domain-specific data for superior instruction-following speech processing.

Principles

• Speech-to-LLM embeddings can be learned using ASR data alone.
• Synthetic data from LLMs and speech synthesis enhances SQA performance.
• Compact models with weaker LLMs can achieve top-tier results.

Method

The method involves a multi-stage training pipeline, integrating SpeechMapper for speech projection and fakACL for synthetic SQA data generation using LLM prompting and SeamlessM4T-large-v2 speech synthesis.

In practice

• Generate synthetic domain-specific data using LLMs and text-to-speech.
• Explore ASR-only data for speech-to-LLM embedding projection.
• Optimize model size and LLM backbone for competitive performance.

Topics

• Speech Processing
• Instruction Following
• Synthetic Data Generation
• Speech-to-LLM Embedding
• SeamlessM4T
• Multi-task Speech Systems

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Evaluation of Automatic Speech Recognition Using Generative Large Language Models — Generative LLMs significantly improve ASR evaluation by aligning better with human perception than traditional WER.
• Deepgram speech-to-text and voice models now available natively on Together AI — Real-time voice agents require integrated STT, LLM, and TTS on a single platform to minimize latency and operational complexity.
• Combining Real and Synthetic Speech for ASR Adaptation in Brazilian Portuguese — Domain-guided synthetic speech effectively augments real data for ASR adaptation in specialized, low-resource scenarios.
• NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR — NIM4-ASR optimizes LLM-based ASR for efficiency and robustness by redesigning multi-stage training and integrating production-ready features.
• A Practical Evaluation Method for Long-Form Simultaneous Speech-to-Speech Translation — A new evaluation method for long-form SimulS2ST uses ASR, forced alignment, and sentence embeddings to measure latency and quality.
• Beyond Cascaded Pipelines: Building a Native Spoken Language Model Prototype — Native Spoken Language Models (SLMs) overcome cascaded pipeline limitations by directly processing audio, preserving nuance, reducing errors, and lowering latency.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.