Neural Speaker Diarization via Multilingual Training: Evaluation on Low-Resource Nepali-Hindi Speech

2026-06-21 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Speech Processing · Depth: Expert, quick

Summary

A study investigates neural speaker diarization for low-resource Nepali-Hindi speech using a multilingual training approach. Speaker diarization, which identifies "who spoke when," typically performs poorly for underrepresented languages due to limited annotated data. Researchers compared two modern architectures, EEND with encoder-decoder attractors (EEND-EDA) and EEND with Perceiver-based attractors (DiaPer), trained on a corpus combining English (LibriSpeech), diverse speakers (VoxCeleb), and collected Nepali and Hindi audio. This setup aimed to reduce language bias and promote cross-lingual generalization. Evaluated across 2-speaker, 3-speaker, 4-speaker, and mixed-speaker scenarios on LibriSpeech, VoxCeleb, and Nepali-Hindi (NeHi) test sets, DiaPer achieved stronger overall performance than EEND-EDA. Specifically, DiaPer obtained DERs of 3.28%, 2.02%, 4.05%, and 4.76% on NeHi 2-speaker, 3-speaker, 4-speaker, and mixed-speaker settings, respectively, demonstrating the viability of Perceiver-based end-to-end neural diarization for low-resource multilingual speech processing.

Key takeaway

For Machine Learning Engineers developing speech applications for underrepresented languages, this research indicates that multilingual training is a viable strategy to overcome data scarcity. You should consider Perceiver-based end-to-end neural diarization (DiaPer) architectures, as they demonstrated superior performance, especially in challenging multi-speaker scenarios. Integrating diverse language corpora, including high-resource and low-resource data, into your training regimen can significantly improve diarization accuracy for languages like Nepali and Hindi.

Key insights

Multilingual training with Perceiver-based attractors significantly improves speaker diarization for low-resource languages like Nepali-Hindi.

Principles

• Multilingual training reduces language bias.
• Cross-lingual generalization is achievable for diarization.
• Perceiver-based attractors enhance multi-speaker performance.

Method

Train EEND models (EEND-EDA, DiaPer) on a multilingual corpus combining high-resource (English) and low-resource (Nepali-Hindi) speech to improve diarization for underrepresented languages.

In practice

• Apply DiaPer for low-resource speech diarization.
• Combine diverse language datasets for training.
• Evaluate on varied speaker count scenarios.

Topics

• Speaker Diarization
• Multilingual Training
• Low-Resource Languages
• Neural Networks
• Perceiver Models
• Speech Processing

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• An Investigation Into Various Approaches For Bengali Long-Form Speech Transcription and Bengali Speaker Diarization — This paper presents a multistage approach for Bengali long-form speech transcription and speaker diarization, addressing the challenges of this low-resource language for the "DL Sprint 4.0" Kaggle competitions.
• What is a Speaker Diarization Engine and how is it used in NLP? — Speaker diarization segments audio by speaker, enabling advanced NLP applications by attributing speech.
• Towards Truly Multilingual ASR: Generalizing Code-Switching ASR to Unseen Language Pairs — Code-switching ASR generalization to unseen language pairs remains limited despite model merging and domain generalization efforts.
• L-Proto: Language-Aware Episodic Prototypical Training for Multilingual Speaker Verification — L-Proto improves multilingual speaker verification by using language-consistent episodic training to disentangle speaker identity from linguistic characteristics.
• Echo: A Joint-Embedding Predictive Architecture for Speaker Diarization and Speech Recognition in a Shared Latent Space — A single 25M-parameter ViT encoder can jointly perform speaker diarization, speech recognition, and source separation in a shared latent space.
• Multilingual Multi-Speaker Unit Vocoders: A Systematic Analysis of Discrete Speech Representations — Cluster size and conditioning strategies are critical for multilingual multi-speaker unit vocoder performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.