NAVER LABS Europe Submission to the Instruction-following 2026 Short Track

2026-06-29 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, short

Summary

NAVER LABS Europe's submission to the IWSLT 2026 instruction-following speech processing short track details a system jointly performing ASR, ST, and SQA from English speech into Chinese, Italian, and German. Building on their previous top-ranked submission, the team updated its multi-stage training pipeline. Key improvements include replacing the speech projector with SpeechMapper, a method learning speech-to-LLM embedding using ASR-only data. Additionally, they introduced fakACL, a synthetic SQA dataset of artificially generated scientific presentations, created by prompting an LLM and synthesizing speech with Seamless. This combination yields a more compact model that outperforms last year's best system, despite relying on a weaker LLM backbone.

Key takeaway

For NLP Engineers developing multi-modal speech systems, consider integrating SpeechMapper and synthetic domain-specific data generation. This approach can yield superior performance with more compact models and less powerful LLMs, potentially reducing inference costs and deployment complexity for your instruction-following applications.

Key insights

Combining SpeechMapper and synthetic data improves multi-modal speech processing with compact, weaker LLMs.

Principles

• Multi-stage training pipelines can be optimized for efficiency.
• Synthetic data generation enhances domain-specific model performance.
• Efficient speech projection improves LLM integration for speech tasks.

Method

The method updates a multi-stage training pipeline by replacing the speech projector with SpeechMapper, which uses ASR-only data. It also introduces fakACL, a synthetic SQA dataset generated by prompting an LLM, segmenting talks, and synthesizing speech with Seamless.

In practice

• Utilize SpeechMapper for speech-to-LLM embedding projection.
• Generate synthetic SQA datasets using LLMs and speech synthesis.
• Design compact models that leverage weaker LLM backbones.

Topics

• Speech Processing
• ASR
• Speech Translation
• SQA
• SpeechMapper
• Synthetic Data

Best for: Research Scientist, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• Combining Real and Synthetic Speech for ASR Adaptation in Brazilian Portuguese — Domain-guided synthetic speech effectively augments real data for ASR adaptation in specialized, low-resource scenarios.
• Half hour of labeling power: Can we beat GPT? — LLMs can create high-quality datasets to train smaller, more accurate, private models for specific business needs.
• Efficient ASR Training with Conversations that Never Happened — LLM-generated synthetic conversations, synthesized via TTS, effectively augment scarce real data for ASR training.
• OpenBMB / VoxCPM — VoxCPM2's tokenizer-free, diffusion autoregressive architecture delivers highly natural, multilingual, and controllable TTS with advanced cloning.
• Voxtral Realtime — Voxtral Realtime delivers offline-quality ASR with sub-second latency via a natively streaming, end-to-end trained architecture.
• Multimodal LLMs are not all you need for Pediatric Speech Language Pathology — Fine-tuned Speech Representation Models with data augmentation outperform LLMs for pediatric Speech Sound Disorder classification.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.