Multilingual Long-Form Speech Instruction Following: KIT’s Submission to IWSLT 2026

2026-06-29 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

KIT's submission to the IWSLT 2026 Instruction Following Track addresses challenges in multilingual long-form speech instruction following, particularly with new and unknown tasks. Their approach utilizes a general data augmentation pipeline that transforms short-form corpora into long-form training data. This pipeline involves segment concatenation, LLM-based label generation, and cross-lingual translation, generating over 1 million instances across six distinct tasks and four languages. The team identified that while likelihood-based re-ranking is effective for Automatic Speech Recognition (ASR), it systematically degrades performance on semantic tasks by favoring outputs from segmented audio processing over holistic long-form inference. This specific failure mode is effectively resolved by integrating likelihood re-ranking with Minimum Bayes Risk (MBR) decoding.

Key takeaway

For NLP Engineers developing multilingual instruction-following systems, consider KIT's data augmentation and decoding strategy. If your models struggle with semantic tasks in long-form speech, especially when using likelihood-based re-ranking, integrate Minimum Bayes Risk (MBR) decoding. This approach can prevent performance degradation caused by segmented audio processing, ensuring more robust and accurate instruction following across diverse languages and tasks.

Key insights

Combining data augmentation with MBR decoding improves multilingual long-form speech instruction following.

Principles

• Data augmentation can create long-form training data from short corpora.
• Likelihood re-ranking can degrade semantic tasks in long-form inference.
• MBR decoding mitigates segmented audio processing issues.

Method

A data augmentation pipeline converts short-form corpora to long-form training data via segment concatenation, LLM-based label generation, and cross-lingual translation, then combines likelihood re-ranking with MBR decoding.

In practice

• Use segment concatenation for long-form data generation.
• Apply LLM-based label generation for diverse tasks.
• Integrate MBR decoding for semantic task robustness.

Topics

• Multilingual Speech Processing
• Instruction Following
• Data Augmentation
• Large Language Models
• Minimum Bayes Risk Decoding
• IWSLT 2026

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Non-Autoregressive Minimum Bayes' Risk Decoding for Fast Speech Recognition — NAR-MBR decoding significantly improves non-autoregressive speech recognition accuracy while maintaining high speed.
• Beyond Bilingual Transfer: Multilingual Code-Switching in Instruction Tuning — Multilingual code-switching instruction tuning improves LLM performance beyond bilingual settings.
• Low-resource Language Discrimination Towards Chinese Dialects with Transfer learning and Data Augmentation — Transfer learning and data augmentation effectively overcome low-resource challenges in Chinese dialect discrimination.
• Structured Inference with Large Language Gibbs — Large Language Gibbs uses LLM conditionals for iterative, bias-free structured probabilistic inference.
• L-Proto: Language-Aware Episodic Prototypical Training for Multilingual Speaker Verification — L-Proto improves multilingual speaker verification by using language-consistent episodic training to disentangle speaker identity from linguistic characteristics.
• Data Augmentations for Data-Constrained Language Model Pretraining — Data augmentations effectively mitigate overfitting in data-constrained language model pretraining, enabling productive multi-epoch training.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.