AITamilDialect@DravidianLangTech 2026: Zero-Shot Whisper and Wav2Vec2 Embedding-Based Tamil Speech Recognition and Dialect Classification.

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing & Speech Technology · Depth: Advanced, medium

Summary

A shared-task system for Tamil speech processing, covering both Automatic Speech Recognition (ASR) and dialect classification, was introduced at DravidianLangTech 2026. The system utilizes the Whisper Large-v3 multilingual model in a zero-shot setting for ASR, without task-specific fine-tuning. For dialect classification, a pre-trained Wav2Vec2 model extracts acoustic features, which are then pooled into utterance-level representations and fed into an XGBoost model for four-way dialect prediction. Experiments on 579 Tamil speech samples resulted in a Word Error Rate (WER) of 0.61 for ASR, indicating the inherent difficulty of dialectal ASR in low-resource environments. The dialect classification achieved an accuracy of 0.49 and a macro F1 score of 0.41, with notable confusion between dialect classes. This system, based purely on standard pre-trained models, establishes a replicable benchmark for future multilingual speech representation evaluations in low-resource Tamil scenarios.

Key takeaway

For NLP Engineers working on low-resource language speech processing, this research highlights the current limitations and benchmark for Tamil. You should consider zero-shot Whisper Large-v3 for ASR and Wav2Vec2 with XGBoost for dialect classification as a baseline. Be aware of the 0.61 WER and 0.49 accuracy, indicating a need for further adaptation or stronger models to improve robustness and reduce dialect confusion.

Key insights

Zero-shot Whisper and Wav2Vec2 embeddings offer a replicable benchmark for Tamil ASR and dialect classification in low-resource settings.

Principles

• Low-resource languages challenge speech tech.
• Dialectal variation complicates ASR.
• Pre-trained models set benchmarks.

Method

The system uses Whisper Large-v3 zero-shot for ASR. For dialect classification, Wav2Vec2 extracts acoustic features, pooled into utterance-level representations, then fed to an XGBoost model for four-way prediction.

In practice

• Use Whisper Large-v3 for ASR.
• Employ Wav2Vec2 for acoustic features.
• Apply XGBoost for dialect prediction.

Topics

• Tamil Speech Recognition
• Dialect Classification
• Whisper Large-v3
• Wav2Vec2 Embeddings
• Low-Resource Languages
• XGBoost

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• BaltiVoice: A Speech Corpus and Fine-tuned Whisper ASR System for the Balti Language — Fine-tuning pre-trained ASR models with modest, newly created corpora can establish functional speech recognition for low-resource languages.
• ArtNet: A JEPA-Like Articulatory Predictive Framework for Robust Zero-Shot Phoneme Recognition — ArtNet uses articulatory feature prediction and VIB to achieve robust zero-shot phoneme recognition across languages.
• dialect2vec: A vector-based method for dialectal transcription of Portuguese from ALiB questionnaires — dialect2vec uses byte-level encoding of IPA sequences to model dialectal variation, outperforming subword models.
• Low-Burden Data Augmentation for Dysarthric ASR via Zero-Shot Voice Cloning — Zero-shot voice cloning effectively augments dysarthric speech data, improving ASR performance and bypassing costly data collection bottlenecks.
• Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India — Real-world ASR performance in India requires unscripted data, orthographically-informed metrics, and regional analysis to overcome current benchmark limitations.
• Hard to Be Heard: Phoneme-Level ASR Analysis of Phonologically Complex, Low-Resource Endangered Languages — Data scarcity, not phonological complexity, primarily drives ASR errors in low-resource, typologically complex languages.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.