An Omnilingual-ASR-Based Speech-LLM System for the 2nd MLC-SLM Challenge

2026-07-14 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

A cascaded diarization-then-recognition system was submitted to Task 1 of the 2nd MLC-SLM Challenge, achieving a macro tcpMER of 29.27% on the Development set and 50.23% on the Evaluation set. This system integrates DiariZen-Large-s80 (WavLM-Large) for segmentation, CAM++ embedding-based two-speaker clustering, and a LoRA-adapted omniASR LLM 7B v2 recognizer, operating without oracle segmentation or speaker labels during testing. The authors highlight two critical engineering decisions: embedding-based speaker clustering significantly outperforms an end-to-end alternative relying solely on ASR turn markers. Additionally, while intended to improve diarization recall, overlap-aware segmentation boosts tcpMER because it transcribes overlapped speech twice, impacting the metric.

Key takeaway

For NLP Engineers developing omnilingual ASR systems, you should prioritize embedding-based speaker clustering over simpler ASR turn marker assignments to improve diarization accuracy. When designing your system, consider integrating LoRA-adapted LLMs like omniASR LLM 7B v2 for robust recognition across diverse languages. Be aware that overlap-aware segmentation, while beneficial for recall, will transcribe overlapped speech twice, potentially affecting your tcpMER metric.

Key insights

A cascaded speech-LLM system for omnilingual ASR leverages embedding-based clustering and LoRA adaptation for robust performance.

Principles

• Embedding-based clustering improves speaker assignment.
• Overlap-aware segmentation can increase tcpMER.
• LoRA adaptation enhances omnilingual ASR.

Method

The system uses DiariZen-Large-s80 for segmentation, CAM++ embedding-based two-speaker clustering, and a LoRA-adapted omniASR LLM 7B v2 for recognition, without oracle labels.

In practice

• Implement CAM++ embedding for speaker clustering.
• Consider LoRA for omnilingual ASR adaptation.
• Evaluate tcpMER impact of overlap-aware segmentation.

Topics

• Omnilingual ASR
• Speech-LLM Systems
• Speaker Diarization
• LoRA Adaptation
• CAM++ Embeddings
• MLC-SLM Challenge

Best for: Research Scientist, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• Diarization-Guided Qwen-ASR Adaptation for Multilingual Two-Speaker Conversational Speech — A diarization-guided, multi-stage ASR adaptation system significantly improves multilingual two-speaker conversational speech recognition.
• MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognition — MoLGE uses language grouping and LoRA to efficiently scale multilingual ASR, overcoming capacity dilution.
• Gradient-Based Speech-to-Text Alignment for Any ASR Model: From CTC to Speech LLMs — A universal gradient-based method aligns speech-to-text for any ASR model, including LLMs, with high precision and no retraining.
• NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR — NIM4-ASR optimizes LLM-based ASR for efficiency and robustness by redesigning multi-stage training and integrating production-ready features.
• DLRG@DravidianLangTech 2026: Dual-Purpose Whisper Adaptation for Tamil Dialect Identification and Dialectal Speech Recognition — A Tamil-adapted Whisper Medium model can effectively unify dialect identification and dialectal ASR tasks.
• Improving quality and robustness in LLM-based text-to-speech systems — LLM-based TTS quality improves through targeted techniques addressing accent leakage, expressiveness, and reliability.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.