ShahiEmotion: A Benchmark Dataset for Punjabi Shahmukhi Emotion Detection

2026-06-26 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, medium

Summary

ShahiEmotion is a newly introduced benchmark dataset designed for Punjabi Shahmukhi emotion detection, a text classification task severely under-resourced for supervised evaluation. The dataset comprises 30379 sentence-level instances, each annotated with one of seven emotion categories: sadness, surprise, happiness, anger, neutral, fear, and disgust. It specifically addresses challenges inherent in low-resource settings, including script-specific issues, lexical variation, and significant class imbalance. Baseline results were established using several pretrained transformer-based models, including multilingual BERT, multilingual DistilBERT, XLM-RoBERTa, and Urdu RoBERTa, fine-tuned for sentence-level classification. XLM-RoBERTa demonstrated the strongest performance, achieving 77.95% accuracy, 58.47% macro-F1, and 77.60% weighted-F1 on the test set. This dataset and its accompanying evaluation protocol are intended to foster future research in Punjabi Shahmukhi emotion analysis and low-resource Natural Language Processing.

Key takeaway

For NLP Engineers and AI Scientists developing emotion detection systems for low-resource languages, particularly Punjabi Shahmukhi, you should utilize the ShahiEmotion dataset. This new benchmark offers 30379 annotated instances across seven emotions, providing a crucial resource where none existed. Consider fine-tuning XLM-RoBERTa, which achieved 77.95% accuracy, as a strong baseline. Your efforts should also focus on strategies to mitigate class imbalance and script-specific challenges inherent in such datasets to improve model performance.

Key insights

ShahiEmotion provides a critical benchmark dataset for Punjabi Shahmukhi emotion detection, a severely under-resourced NLP task.

Principles

• Low-resource languages need dedicated datasets.
• Transformer models can establish strong baselines.
• Class imbalance is a key challenge in low-resource NLP.

Method

Emotion detection is formulated as a sentence-level classification task, fine-tuning pretrained transformer models (multilingual BERT, DistilBERT, XLM-RoBERTa, Urdu RoBERTa) with standard cross-entropy loss.

In practice

• Use XLM-RoBERTa for Shahmukhi emotion tasks.
• Address class imbalance in low-resource datasets.
• Develop resources for under-represented languages.

Topics

• Punjabi Shahmukhi
• Emotion Detection
• Low-Resource NLP
• Benchmark Datasets
• Transformer Models
• XLM-RoBERTa

Best for: Research Scientist, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• How to Fine-Tune an SLM for Emotion Recognition — Fine-tuning SLMs for fine-grained emotion recognition on imbalanced datasets requires a multi-technique approach to data balancing and loss weighting.
• A Community Roadmap for Including Kapampángan in Modern AI Research — Community-driven data creation is essential for including underrepresented languages in AI research and applications.
• Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India — Real-world ASR performance in India requires unscripted data, orthographically-informed metrics, and regional analysis to overcome current benchmark limitations.
• NextMotionQA: Benchmarking and Judging Human Motion Understanding with Vision-Language Models — NextMotionQA is a new benchmark for human motion understanding, revealing VLM limitations in fine-grained analysis.
• SICI: A Semantic-Pragmatic Complexity Index Reveals Regime Shifts in LLM Stance Detection — LLM stance detection errors exhibit predictable regime shifts tied to semantic-pragmatic complexity, leading to distinct error patterns.
• UrduMMLU: A Massive Multitask Benchmark for Urdu Language Understanding — UrduMMLU reveals current LLMs have uneven Urdu knowledge, particularly for regionally grounded content, despite strong performance from top models.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.