FBK’s Long-form SpeechLLMs for IWSLT 2026 Instruction Following

2026-06-29 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

FBK's SpeechLLM systems were developed for the IWSLT 2026 Instruction Following shared task, addressing both short-form and long-form speech under constrained settings. For the short track, the systems achieved strong performance on MCIF, with a SIFS score of 2.0708. In the long track, researchers investigated three speech segmentation strategies, introducing the HIFS score to manage unstable long-form generation. Experimental results indicated that fixed 30-second segmentation provided the most robust long-form performance, yielding the highest HIFS score of 2.0663. Further analysis revealed that hallucination primarily manifests as repetitive insertions, substantially affecting ASR and SSUM, while short-form capabilities largely persist after long-form extension.

Key takeaway

For NLP engineers developing long-form speech instruction following systems, you should prioritize fixed 30-second speech segmentation to achieve robust performance, as demonstrated by FBK's SpeechLLMs. Be aware that hallucination often appears as repetitive insertions, which can significantly impact ASR and summarization outputs, requiring careful post-processing or model fine-tuning to maintain output quality.

Key insights

FBK's SpeechLLMs for instruction following demonstrate the impact of speech segmentation on long-form performance and characterize hallucination.

Principles

• Long-form speech generation requires robust segmentation.
• Hallucination in long-form often involves repetitive insertions.
• Short-form capabilities can be retained in long-form extensions.

Method

Investigated three speech segmentation strategies for long-form speech instruction following, introducing the HIFS score to account for unstable long-form generation.

In practice

• Implement fixed 30-second segmentation for robust long-form speech.
• Design post-processing to detect repetitive insertions in ASR/SSUM.

Topics

• SpeechLLMs
• Instruction Following
• Speech Segmentation
• Hallucinations
• IWSLT 2026
• Automatic Speech Recognition
• Speech Summarization

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• FBK's Long-form SpeechLLMs for IWSLT 2026 Instruction Following — Fixed 30-second segmentation provides robust long-form SpeechLLM performance, mitigating hallucination effects on ASR and SSUM.
• Detecting Hallucinations in SpeechLLMs at Inference Time Using Attention Maps — Attention-derived metrics enable efficient, inference-time hallucination detection in SpeechLLMs without gold-standard outputs.
• Analyzing Error Propagation in Korean Spoken QA with ASR-LLM Cascades — ASR-stage information loss, especially single-character errors in Korean, is the primary bottleneck for ASR-LLM cascade performance.
• Expert Masking for More Efficient Expert Offloading — Fine-tuning can degrade existing factual knowledge, but strategies like parameter restriction or self-distillation can mitigate this.
• LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Models — Linguistic tone systematically influences VLM hallucination, with varying effects on frequency and intensity across models and tasks.
• Speech-Guided Multimodal Learning for Vocal Tract Segmentation in Real-Time MRI — Multimodal supervision during training can enhance single-modality inference for challenging medical image segmentation tasks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.