Wise@DravidianLangTech 2026: Dialect-Aware Tamil Speech Classification and Recognition via Cross-Pipeline Embedding Transfer

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

The Wise system, presented at Wise@DravidianLangTech 2026, addresses dialect-based speech processing in Tamil, focusing on four-way dialect region classification and dialectal Tamil ASR. Audio undergoes loudness normalization and neural denoising. For classification, the system combines multilingual and Tamil-pretrained Wav2Vec2 backbones with various temporal pooling strategies and fine-tuning. Its top configuration, utilizing learned attentive pooling with partial fine-tuning and a differentially trained MLP head, achieved a macro F1 of 0.79, securing 1st place with a 0.26-point margin. For ASR, Wise proposes two novel dialect-conditioned Whisper architectures—residual injection and cross-attention—which inject dialect embeddings from the classifier. A vanilla Whisper-Tamil baseline was also evaluated. The best ASR model achieved a WER of 0.90, placing 8th in the shared task.

Key takeaway

For NLP Engineers developing speech recognition systems for dialectal or low-resource languages, this work highlights the value of integrating dialect classification. You should explore cross-pipeline embedding transfer, specifically injecting dialect embeddings from a robust classifier into your ASR pipeline. This approach, demonstrated by the Wise system's novel Whisper architectures, can significantly improve dialect-aware ASR performance, even if initial classification results are stronger than ASR. Evaluate this method to enhance your models' sensitivity to linguistic variations.

Key insights

Integrating dialect classification embeddings into ASR pipelines enhances dialect-aware speech processing for low-resource languages like Tamil.

Principles

• Cross-pipeline embedding transfer improves downstream tasks.
• Audio preprocessing ensures consistent quality.
• Partial fine-tuning with attentive pooling boosts classification.

Method

Preprocess audio with denoising, classify dialect using fine-tuned Wav2Vec2, then inject these dialect embeddings into Whisper ASR via residual injection or cross-attention for dialect-conditioned recognition.

In practice

• Implement neural denoising for audio preprocessing.
• Utilize Wav2Vec2 for dialect classification tasks.
• Inject dialect embeddings into ASR for improved accuracy.

Topics

• Tamil Speech Processing
• Dialect Classification
• Automatic Speech Recognition
• Wav2Vec2
• Whisper Models
• Embedding Transfer

Best for: Research Scientist, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects — PolySpeech-100 reveals E2E Speech-LLMs excel on dialects, but open-source models struggle with low-resource languages, and CoT prompting can degrade performance.
• A Comparative Study of Pretrained Transformer Models for Quranic ASR: Speech Representations, Label Formats, and Dataset Composition — Fine-tuning pretrained Transformers with specific speech representations and label formats significantly improves Quranic ASR accuracy and efficiency.
• Speech-Driven End-to-End Language Discrimination towards Chinese Dialects — Speech-driven features, integrating MFCC and word embeddings through a CNN, significantly improve Chinese dialect discrimination.
• Speech Emotion Recognition Leveraging OpenAI's Whisper Representations and Attentive Pooling Methods — Whisper's pre-trained ASR representations, combined with attention-based pooling, significantly enhance Speech Emotion Recognition.
• Towards Truly Multilingual ASR: Generalizing Code-Switching ASR to Unseen Language Pairs — Code-switching ASR generalization to unseen language pairs remains limited despite model merging and domain generalization efforts.
• Improving low-resource ASR using bilingual fine-tuning with language identification: a cross-linguistic evaluation — Bilingual fine-tuning enhances low-resource ASR, particularly when language identification is accurate or explicitly guided during inference.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.