NAVER LABS System Re-implementation for the IWSLT 2026 Instruction-Following Task

2026-07-08 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, long

Summary

NAVER LABS has re-implemented its IWSLT 2025 instruction-following pipeline for the IWSLT 2026 Shared Task, adapting it to mandated components: SeamlessM4T-v2-large as the speech encoder and Qwen3-4B-Instruct as the LLM backbone. This open-source re-implementation preserves the original three-stage approach, involving projector alignment, text-only LoRA pre-training, and multimodal merging. The team also generated 100k synthetic instruction-following examples across ten speech-centric task types, suitable for Stage 3 fine-tuning. Their primary Stage 3 model achieved a COMET score of 0.781 on EN–ZH speech translation and a BERTScore-F1 of 0.346 on English SQA on the MCIF benchmark. This system demonstrates consistent improvements over projector-only baselines, with code, training scripts, and the synthetic dataset publicly available.

Key takeaway

For Machine Learning Engineers developing multimodal speech LLMs for instruction-following tasks, you should consider adopting a staged training approach. This re-implementation demonstrates that combining projector alignment, text-only LoRA pre-training, and multimodal merging with specific components like SeamlessM4T-v2-large and Qwen3-4B-Instruct yields competitive results. Leveraging the released 100k synthetic dataset can significantly enhance your model's performance on diverse speech-centric tasks, particularly for IWSLT 2026 participants.

Key insights

A three-stage pipeline with specific encoder/LLM and synthetic data achieves strong multimodal instruction-following performance.

Principles

• Multimodal LLMs benefit from staged training.
• Synthetic data enhances instruction-following.
• LoRA rank impacts task-specific performance.

Method

The system uses a three-stage pipeline: projector alignment, text-only LoRA pre-training, and joint multimodal fine-tuning of projector and LoRA adapters.

In practice

• Use SeamlessM4T-v2-large with Qwen3-4B-Instruct.
• Generate 100k synthetic speech-centric examples.
• Fine-tune LoRA with rank 8 for SQA tasks.

Topics

• Multimodal LLMs
• Instruction Following
• Speech Translation
• Spoken Question Answering
• LoRA Fine-tuning
• Synthetic Data

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Building LLMs from the Ground Up: A 3-hour Coding Workshop — Building LLMs from scratch, including data preparation, architecture, pre-training, and fine-tuning, enhances fundamental understanding.
• Are you speaking my languages? On spoken language adherence in multimodal LLMs — Multimodal LLMs require soft prompting and specific strategies to ensure spoken language adherence in ASR, preventing transcription errors.
• AlignAtt4LLM: Fast AlignAtt for Decoder-Only LLMs at IWSLT 2026 Simultaneous Speech Translation Task — Adapting AlignAtt for decoder-only LLMs enables simultaneous speech translation by re-engineering attention mechanisms.
• VCIFBench: Evaluating Complex Instruction Following for Video Understanding — MLLMs struggle with complex, constraint-rich video instruction following, highlighting a critical evaluation gap.
• English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Training — Multilingual post-training significantly improves LLM performance across languages and tasks, outperforming English-only approaches.
• Announcing the ICLR 2026 Outstanding Papers — Transformer efficiency stems from succinct concept encoding, while LLMs struggle with multi-turn conversations due to training data mismatches.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.