Diarization-Guided Qwen-ASR Adaptation for Multilingual Two-Speaker Conversational Speech

2026-07-09 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

A self-designed system for Task 1 of the MLC-SLM 2026 Challenge addresses multilingual two-speaker conversational speech. This system integrates a modular speaker diarization front end with a Qwen3-ASR-1.7B recognizer adapted for the challenge. The diarization component performs voice activity detection, subsegment generation, CAMPPlus speaker embedding extraction, two-speaker spectral clustering, and RTTM-based audio segmentation. Speaker-attributed segments are then grouped by language or region for decoding by the adapted ASR model. ASR adaptation involves three stages: supervised full fine-tuning on official training data, LoRA fine-tuning using synthetic speech from a three-pipeline TTS framework, and GRPO reinforcement learning with WER/CER rewards and penalties for hallucination, repetition, and length deviation. The full system achieved an average tcpMER of 23.70 on the development set, a 6.83 absolute point reduction compared to the baseline Qwen-ASR-1.7B, and 17.97 on the final evaluation set. Ablation studies confirmed supervised fine-tuning provided the most significant gain, with synthetic-speech LoRA and reinforcement learning enhancing robustness.

Key takeaway

For Machine Learning Engineers developing multilingual ASR systems for conversational speech, you should consider a modular, diarization-guided approach combined with multi-stage ASR adaptation. Prioritize supervised fine-tuning on relevant data for the largest performance gains. Subsequently, integrate synthetic speech LoRA adaptation and GRPO reinforcement learning to further enhance model robustness and reduce errors like hallucination and repetition, particularly in two-speaker conversational scenarios.

Key insights

A diarization-guided, multi-stage ASR adaptation system significantly improves multilingual two-speaker conversational speech recognition.

Principles

• Modular diarization front ends enhance ASR.
• Multi-stage ASR adaptation improves performance.
• Synthetic data and RL boost ASR robustness.

Method

The system uses VAD, subsegment generation, CAMPPlus embeddings, spectral clustering, and RTTM segmentation for diarization, followed by multi-stage ASR adaptation.

In practice

• Combine diarization with ASR for multi-speaker.
• Use LoRA with synthetic data for ASR adaptation.
• Apply GRPO reinforcement learning for ASR refinement.

Topics

• Speaker Diarization
• Automatic Speech Recognition
• Qwen-ASR
• LoRA Fine-tuning
• Reinforcement Learning
• Multilingual Speech

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• An Omnilingual-ASR-Based Speech-LLM System for the 2nd MLC-SLM Challenge — A cascaded speech-LLM system for omnilingual ASR leverages embedding-based clustering and LoRA adaptation for robust performance.
• Polyglot-Lion: Efficient Multilingual ASR for Singapore via Balanced Fine-Tuning of Qwen3-ASR — Balanced fine-tuning of moderate-scale ASR models yields cost-effective, high-performance multilingual speech recognition.
• Towards Truly Multilingual ASR: Generalizing Code-Switching ASR to Unseen Language Pairs — Code-switching ASR generalization to unseen language pairs remains limited despite model merging and domain generalization efforts.
• When Synthetic Speech Is All You Have: Better Call GRPO — GRPO, a critic-free reinforcement learning method, significantly outperforms SFT for LLM-based ASR adaptation using synthetic speech.
• Neural Speaker Diarization via Multilingual Training: Evaluation on Low-Resource Nepali-Hindi Speech — Multilingual training with Perceiver-based attractors significantly improves speaker diarization for low-resource languages like Nepali-Hindi.
• UtterTune: LoRA-Based Target-Language Pronunciation Edit and Control in Multilingual Text-to-Speech — LoRA-based adaptation enables precise phoneme and pitch accent control in multilingual LLM-TTS for specific languages like Japanese.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.