ShefFriday at SemEval-2026 Task 9: LLM-Based Annotation Methods for Detecting Multilingual, Multicultural and Multievent Online Polarisation

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Natural Language Processing · Depth: Expert, quick

Summary

ShefFriday presented its findings for SemEval-2026 Task 9, employing an LLM-as-an-annotator strategy across all three subtasks. This approach simulated data annotation using large language models, creating 30 LLM annotators via persona injection, also known as sociodemographic prompting. The team experimented with various annotation aggregation methods, including Dawid-Skene and MACE, and found majority voting performed best for subtasks 2 and 3. To enhance annotator response variability, hatefulness detection was used as a proxy for identifying online polarization. This reframing proved effective for binary classification of polarization but less so for finer-grained detection. While their unsupervised method did not achieve the top ranks of supervised approaches, the work offers valuable insights into persona-based prompting and the challenge of high intra-model agreement among LLM annotators.

Key takeaway

For NLP Engineers designing data annotation pipelines, consider LLM-as-an-annotator for binary classification tasks, especially when data scarcity is a concern. Your approach should incorporate persona injection to diversify LLM responses and evaluate aggregation methods like majority voting, which proved effective for subtasks 2 and 3 in SemEval-2026 Task 9. Be mindful of high intra-model agreement, which may limit finer-grained detection capabilities.

Key insights

LLM-as-annotator with persona injection effectively detects binary polarization but struggles with finer granularity and high intra-model agreement.

Principles

• Persona injection diversifies LLM annotator responses.
• Task reframing aids binary classification.
• LLM annotators exhibit high intra-model agreement.

Method

Create 30 LLM annotators via persona injection. Reframe polarization detection as hatefulness detection for variability. Aggregate annotations using methods like Dawid-Skene or majority voting for classification.

In practice

• Apply persona injection for diverse LLM annotations.
• Use majority voting for LLM annotation aggregation.
• Reframe complex tasks for binary classification.

Topics

• LLM-Based Annotation
• SemEval-2026 Task 9
• Online Polarization
• Persona Injection
• Annotation Aggregation
• Hatefulness Detection

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• LLM-based Detection of Manipulative Political Narratives — LLM-driven pipelines can effectively identify and cluster manipulative political narratives in raw social media data.
• I fine-tuned an LLM to be C-3PO to test which training data format works best for persona injection [P] — First-person statements are most effective for LLM persona injection, outperforming chat demos and synthetic docs.
• Polarization by Default: Auditing Recommendation Bias in LLM-Based Content Curation — LLM content curation amplifies polarization and exhibits consistent negative sentiment, with varying provider-specific biases.
• When Does Delegation Beat Majority? A Delegation-Based Aggregator for Multi-Sample LLM Inference — Delegation-based aggregation using semantic entropy and reasoning geometry significantly improves LLM multi-sample inference over majority voting.
• Prompt Injection in Automated Résumé Screening with Large Language Models: Single and Multi-Injection Settings — Prompt injection can manipulate LLM-based résumé screening, but its effectiveness decreases with widespread use and varies with candidate quality.
• LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation — LLMs can generate synthetic data to validate NLP evaluation metrics, replacing costly human annotations.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.