Unibuc-NLP at SemEval-2026 Task 10: Unmasking Conspiracies with Pre-Trained Language Models

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

Unibuc-NLP submitted a system to SemEval-2026 Task 10 (PsyCoMark) Subtask 2, focusing on detecting conspiracy beliefs in Reddit comments. Their approach investigated three modeling paradigms: an embedding-and-classify pipeline using Jina-embeddings-v3, HateBERT, and BERT-Sentiment with classical ML models and 19 readability features; end-to-end fine-tuning of encoder transformers like DeBERTa-v3-base and DistilBERT; and QLoRA fine-tuning of decoder-only models such as Mistral-7B-v0.3 and Qwen3-0.6B. The submitted DeBERTa-v3-base system, utilizing a 128-dimensional classifier, achieved a weighted F1 of 0.74, ranking 29th out of 52 participants. Subsequent analysis revealed that a weighted pooling strategy improved performance to a weighted F1 of 0.78 on the validation split, moving it to rank 8/52, indicating that conspiracy features are distributed across transformer layers.

Key takeaway

For NLP Engineers developing text classification systems, you should investigate weighted pooling strategies for encoder transformers like DeBERTa-v3-base. This approach, which improved F1 by +0.04 in conspiracy detection, suggests that critical features are not confined to the final [CLS] token. Optimizing pooling can significantly enhance model performance, especially when dealing with nuanced linguistic phenomena distributed across different layers. Consider experimenting with various pooling methods beyond standard [CLS] or mean pooling in your next project.

Key insights

Weighted pooling with DeBERTa-v3-base significantly improves conspiracy detection by leveraging features distributed across transformer layers.

Principles

• Conspiracy-relevant features distribute across transformer layers.
• Encoder transformers can achieve strong performance in text classification.

Method

The system explored embedding-and-classify pipelines, end-to-end fine-tuning of encoder transformers, and QLoRA fine-tuning of decoder-only models for conspiracy detection.

In practice

• Use weighted pooling instead of [CLS] for transformer classification.
• Consider DeBERTa-v3-base for conspiracy belief detection.
• Evaluate diverse pooling strategies for transformer-based text classification.

Topics

• Conspiracy Detection
• DeBERTa-v3-base
• Transformer Pooling
• QLoRA
• SemEval-2026
• Text Classification

Best for: Research Scientist, AI Scientist, NLP Engineer, Machine Learning Engineer

Related on AIssential

• mdok-style at SemEval-2026 Task 10: Finetuning LLMs for Conspiracy Detection — Data augmentation and self-training can effectively finetune LLMs for specialized text classification tasks with limited data.
• An End-to-End Hybrid Framework for Rumour Detection in Low-Resources Algerian Dialect — A hybrid framework combining transformer embeddings with classical classifiers enables effective rumour detection in low-resource Algerian dialect.
• Fake News Detection Using Natural Language Processing! — Combining linguistic, syntactic, and semantic features with machine learning achieves high accuracy in fake news detection.
• Rethinking Negative Sampling for Knowledge Distillation in Retrieval, A Distillation Recipe for Small Language Model Search Agents, and More! — Effective information retrieval advancements often stem from rethinking fundamental assumptions and optimizing data representation or model behavior.
• The Geometry of Truth: Detecting LLM Hallucinations with Geometric Algebra — Type II hallucinations are context contradictions, dangerous due to their plausibility and standard similarity metric blindness.
• FineREX: Fine-Tuned NER-RE for Human Smuggling Knowledge Graphs — Domain-specific fine-tuning of LLMs significantly enhances knowledge graph construction quality and efficiency for complex, jargon-heavy legal texts.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.