Hope_Speech_Alchemists@DravidianLangTech 2026: TF-IDF SVM and XLM-RoBERTa with Focal Loss for Hope Speech Detection in Tulu

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Data Science & Analytics · Depth: Expert, short

Summary

The Hope_Speech_Alchemists team submitted a system for hope speech detection in Tulu to the DravidianLangTech@ACL 2026 shared task. Their system addresses two sub-tasks: coarse-grained classification into four categories (Task 1) and fine-grained classification into five categories (Task 2). They compared a traditional TF-IDF + LinearSVC baseline against an XLM-RoBERTa model fine-tuned with minority-class oversampling and Focal Loss. Experiments showed a trade-off: XLM-RoBERTa achieved the best validation Macro-F1 of 0.57 on the coarse-grained task, but the TF-IDF baseline surpassed it on the smaller fine-grained task. This indicates that large pre-trained models may struggle with data scarcity. On the official test set, their system achieved a Macro-F1 of 0.55 for Task 1 and 0.40 for Task 2. The project code is publicly available.

Key takeaway

For NLP Engineers developing models for low-resource languages or fine-grained classification tasks, you should carefully evaluate traditional baselines like TF-IDF SVM. Your large pre-trained models, even with techniques like Focal Loss and oversampling, might struggle with data scarcity, as seen in the Tulu hope speech detection task. Consider starting with simpler models for smaller datasets before investing in complex transformer fine-tuning.

Key insights

Large pre-trained models may underperform simpler baselines on tasks with limited fine-grained data.

Principles

• Data scarcity impacts large model generalization.
• Simpler models can excel on small datasets.
• Focal Loss aids imbalanced classification.

Method

The system fine-tuned XLM-RoBERTa using minority-class oversampling and Focal Loss, comparing it against a TF-IDF + LinearSVC baseline for hope speech detection.

In practice

• Evaluate simpler models for low-resource language tasks.
• Apply Focal Loss for imbalanced classification problems.
• Consider data augmentation for fine-grained tasks.

Topics

• Hope Speech Detection
• Tulu Language
• XLM-RoBERTa
• TF-IDF SVM
• Low-Resource NLP
• Imbalanced Classification

Code references

• meclin2345/Hope_Speech_Alchemists
• meclin2345/Hope

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• mdok-style at SemEval-2026 Task 9: Finetuning LLMs for Multilingual Polarization Detection — Finetuning mid-sized LLMs with QLoRA and augmented data improves multilingual online polarization detection.
• TF-IDF (Term Frequency-Inverse Document Frequency) Explained — TF-IDF quantifies word importance by balancing local frequency with global rarity for text feature extraction.
• "Sorry, I Didn't Catch That": How Speech Models Miss What Matters Most — Speech models fail on high-stakes, short utterances despite low benchmark word error rates, especially for non-English speakers.
• Towards interpretable models for language proficiency assessment: Predicting the CEFR level of Estonian learner texts — Careful feature selection yields interpretable, generalizable models for language proficiency assessment with high accuracy.
• Detecting HIV-Related Stigma in Clinical Narratives Using Large Language Models — LLMs can effectively detect HIV-related stigma in clinical notes, with encoder models outperforming generative models in zero-shot settings.
• I spent 4 days training an AI model to understand my country. It barely works — Low-resource NLP projects face significant challenges in data collection, translation, and accurate labeling, often yielding modest baselines.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.