From Monolingual to Multilingual: Evaluating Mamba for ASR in South African Languages

2026-07-01 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

This work evaluates the Mamba state space model for Automatic Speech Recognition (ASR) across seven South African languages, comparing its performance against a Conformer baseline of similar parameter scale. In monolingual experiments, Mamba achieved recognition accuracy comparable to Conformer while utilizing fewer computational resources and demonstrating faster training times, using 50 hours of speech per language. Both models, however, struggled to generalize to speech significantly longer than their training data. Multilingual training consistently improved ASR performance over monolingual approaches. While adding explicit language-family information via embeddings did not enhance in-domain performance, it significantly improved cross-corpus robustness. Ablation studies in low-resource settings, using 5-hour and 10-hour per-language training data, further confirmed the benefits of language embeddings, revealing they function as task-specific control vectors rather than capturing typological linguistic similarity.

Key takeaway

For NLP Engineers developing ASR systems for low-resource South African languages, consider Mamba models for their computational efficiency and comparable accuracy to Conformer. You should prioritize multilingual training, as it consistently improves performance. Integrate language embeddings, especially in low-resource settings, to enhance cross-corpus robustness. Understand these embeddings act as task-specific control vectors, not linguistic similarity indicators. This approach can optimize resource usage while improving model generalization.

Key insights

Mamba offers Conformer-level ASR accuracy with less compute, and multilingual training with control vectors improves robustness for low-resource languages.

Principles

• Mamba matches Conformer ASR efficiency.
• Multilingual training boosts ASR performance.
• Language embeddings act as control vectors.

Method

The study evaluates Mamba and Conformer models for ASR, using monolingual and multilingual training with language/language-family embeddings and multitask learning (CTC ASR + LID head).

In practice

• Use Mamba for efficient ASR.
• Implement multilingual training.
• Add language embeddings for robustness.

Topics

• Automatic Speech Recognition
• Mamba Model
• Conformer
• South African Languages
• Multilingual ASR
• Language Embeddings
• Low-Resource ASR

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Paza: Introducing automatic speech recognition benchmarks and models for low resource languages — Human-centered design and evaluation are crucial for effective speech models in low-resource, real-world environments.
• Towards Truly Multilingual ASR: Generalizing Code-Switching ASR to Unseen Language Pairs — Code-switching ASR generalization to unseen language pairs remains limited despite model merging and domain generalization efforts.
• Pretrained self-supervised speech models can recognize unseen consonants — Pretrained self-supervised speech models effectively generalize to and accurately recognize typologically rare phonemes, such as click consonants.
• Neural Speaker Diarization via Multilingual Training: Evaluation on Low-Resource Nepali-Hindi Speech — Multilingual training with Perceiver-based attractors significantly improves speaker diarization for low-resource languages like Nepali-Hindi.
• UMA-Split: unimodal aggregation for both English and Mandarin non-autoregressive speech recognition — UMA-Split extends unimodal aggregation to English non-autoregressive ASR by allowing aggregated frames to map to multiple fine-grained tokens.
• Improving low-resource ASR using bilingual fine-tuning with language identification: a cross-linguistic evaluation — Bilingual fine-tuning enhances low-resource ASR, particularly when language identification is accurate or explicitly guided during inference.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.