Annotating Indian Regional Biases using Large Language Models: Evaluation and Analysis

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, medium

Summary

The paper "Annotating Indian Regional Biases using Large Language Models: Evaluation and Analysis" by Panda, Anil, and Shukla, presented at *SEM 2026, investigates the use of open-source Large Language Models (LLMs) for annotating Indian regional biases. The study utilizes the IndRegBias dataset, which contains social media comments in both English and code-mixed formats. Initially, LLMs showed low agreement with human annotations (measured by Cohen's kappa) in a zero-shot setting across various writing styles, including code-mixing and transliteration. However, fine-tuning these LLMs with 50% of the IndRegBias dataset significantly improved annotation agreement. Further evaluation on 500 newly collected social media comments confirmed that the fine-tuned LLMs consistently outperformed their zero-shot counterparts in identifying regional biases. This research highlights both the challenges and the potential of LLMs for bias detection in linguistically diverse, low-resource contexts like India.

Key takeaway

For NLP Engineers developing bias detection systems for Indian languages, you should prioritize fine-tuning open-source LLMs. Zero-shot approaches show low agreement with human annotations for regional biases in code-mixed content. Fine-tuning with 50% of a dataset like IndRegBias significantly enhances annotation accuracy. This improved performance extends to new, diverse social media comments, making fine-tuning crucial for effective and culturally sensitive bias detection tools.

Key insights

Fine-tuning significantly improves LLM performance for annotating Indian regional biases in code-mixed social media text, overcoming zero-shot limitations.

Principles

• LLMs struggle with low-resource languages zero-shot.
• Fine-tuning improves LLM agreement with human annotations.
• Regional biases require context-specific annotation.

Method

The study assessed open-source LLMs on the IndRegBias dataset in zero-shot, then fine-tuned them with 50% of the data, evaluating on the remaining 50% and 500 new comments.

In practice

• Fine-tune LLMs for low-resource bias detection.
• Use IndRegBias for Indian regional bias tasks.
• Evaluate LLMs on code-mixed social media.

Topics

• Indian Regional Bias
• Large Language Models
• Bias Detection
• Code-mixing
• Fine-tuning LLMs
• IndRegBias Dataset

Best for: AI Engineer, Research Scientist, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• IndoBias: A Dual Track Culturally Grounded Benchmark for LLMs Bias Evaluation in Indonesian Languages — Culturally-grounded benchmarks are crucial for evaluating LLM bias in diverse, multilingual contexts like Indonesia.
• Cross-lingual Relation Extraction with Large Language Models: Zero-Shot, Few-Shot, and Fine-Tuned Evaluation on Romanian — QLoRA fine-tuning significantly boosts cross-lingual relation extraction performance for low-resource languages, outperforming prompt-only LLMs.
• Addressing Data Scarcity in Bangla Fake News Detection: An LLM-Based Dataset Augmentation Approach — LLM-based data augmentation significantly improves fake news detection in low-resource languages by generating diverse synthetic data.
• Evaluating Cross-lingual Knowledge Consistency in Code-Mixed vis-a-vis Indian Languages using IndicKLAR — Code-mixing inputs significantly improves large language model knowledge recall in Indian languages, nearly matching English performance.
• Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs — Multilingual LLMs exhibit implicit US-centric and population-driven biases, exacerbated by instruction tuning.
• TalkTag: Fine-Grained Morphosyntactic Error Annotation for Transcribed Speech — TalkTag automates fine-grained morphosyntactic error annotation in spoken transcripts using an LLM, even with scarce data.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.