Truth Gradient at SemEval-2026 Task 10:Conspiracy Belief Detection via Narrative Density and Mean Pooling

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Software Development & Engineering · Depth: Expert, medium

Summary

Truth Gradient's system for SemEval-2026 Task 10 Subtask 2 addresses conspiracy belief detection, proposing that believers exhibit "narrative density" through significantly more psycholinguistic markers per post (Cohen's d = 0.56, p 10⁻⁸⁰). The system employs a DeBERTaV3-large model with mean pooling and a 5-seed probability-averaging ensemble. It achieved a macro F1 of 0.829 on the 77-sample development set and 0.75 on the official test set. The authors recommend using a 5-fold cross-validation estimate (0.734 ± 0.007) as a more reliable performance predictor for low-resource shared tasks. Further analysis showed masking marker spans reduced F1 by 5.3 percentage points, while marker-count fusion recovered 0.9 percentage points. Encoder fine-tuning was identified as the dominant design factor, and belief information peaked at layer 16/24 in the model's mid-stack layers.

Key takeaway

For NLP Engineers developing systems for misinformation detection, this research suggests focusing on "narrative density" as a key psycholinguistic indicator. You should prioritize fine-tuning encoder layers in models like DeBERTaV3-large, as this significantly impacts performance. When evaluating on low-resource datasets, adopt 5-fold cross-validation for more reliable performance estimates, mitigating the risk of overfitting to small development sets.

Key insights

Conspiracy belief correlates with "narrative density" of psycholinguistic markers, detectable by fine-tuned language models.

Principles

• "Narrative density" indicates conspiracy belief.
• 5-fold CV improves reliability for low-resource tasks.
• Encoder fine-tuning is critical for detection.

Method

DeBERTaV3-large with mean pooling and 5-seed probability-averaging ensemble for classification. Use 5-fold cross-validation for robust performance estimation in low-resource settings.

In practice

• Implement DeBERTaV3-large for text classification.
• Apply 5-fold CV for small dataset evaluation.
• Focus fine-tuning on encoder layers for performance.

Topics

• Conspiracy Belief Detection
• Narrative Density
• DeBERTaV3-large
• SemEval-2026
• Cross-validation
• Psycholinguistic Markers

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• mdok-style at SemEval-2026 Task 10: Finetuning LLMs for Conspiracy Detection — Data augmentation and self-training can effectively finetune LLMs for specialized text classification tasks with limited data.
• Layer-Resolved Optimal Transport for Hallucination Detection in NMT and Abstractive Summarization — Optimal Transport detects hallucinations from source disengagement, but not content misrepresentation.
• Misinformation Span Detection in Videos via Audio Transcripts — Pinpointing misinformation within videos via audio transcript spans offers more granular, interpretable detection than video-level classification.
• A Resource for Enthymeme Detection in Controversial Political Discourse — The study of annotator disagreement in enthymeme detection improves model performance and understanding of human inference.
• The Geometry of Truth: Detecting LLM Hallucinations with Geometric Algebra — Type II hallucinations are context contradictions, dangerous due to their plausibility and standard similarity metric blindness.
• Probing Multimodal Large Language Models on Cognitive Biases in Chinese Short-Video Misinformation — MLLMs struggle with short-video misinformation due to cognitive biases like authority and popularity effects, despite multimodal reasoning capabilities.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.