psy detectives at SemEval-2026 Task 10: PsyCoMark – Psycholinguistic Conspiracy Marker Extraction and Detection

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Social Sciences & Behavioral Studies · Depth: Expert, medium

Summary

The "psy detectives" system, presented at SemEval-2026 Task 10 (PsyCoMark), combines interpretable psycholinguistic signals with supervised neural modeling for conspiracy marker extraction and detection. Its approach integrates a marker-derived lexicon and LIWC-style ratio features from span annotations, alongside binary Yes/No transformer baselines from the RoBERTa and DeBERTa families. The system also includes a zero-shot TinyLlama-1.1B baseline for a three-way classification (Yes/No/Can't tell). While marker-only features proved transparent but weak, transformer models demonstrated significantly stronger performance. Specifically, DeBERTa-v3-large achieved a 0.8339 weighted F1 score on the development set and 0.75 weighted F1 on the competition test set. The evaluation also found that marker-driven heuristic relabeling of uncertain instances did not enhance downstream performance. This work establishes a controlled, interpretable, and reproducible reference for future research on integrating span-level psycholinguistic evidence into conspiracy detection.

Key takeaway

For NLP Engineers developing systems for misinformation or conspiracy detection, you should prioritize transformer-based architectures over purely psycholinguistic feature sets. While psycholinguistic markers offer interpretability, models like DeBERTa-v3-large deliver substantially higher performance, achieving 0.75 weighted F1 on test data. Focus your efforts on optimizing neural model configurations for robust detection, as marker-driven heuristic relabeling of uncertain instances proved ineffective. Consider this a strong baseline for integrating advanced neural methods with psycholinguistic insights.

Key insights

Transformer models significantly outperform psycholinguistic marker-only features for conspiracy detection.

Principles

• Psycholinguistic features offer interpretability but limited predictive power.
• Neural models provide superior performance in complex text classification.
• Combining diverse signal types can establish research baselines.

Method

The system integrates a marker-derived lexicon, LIWC-style ratio features, and transformer baselines (RoBERTa, DeBERTa) for binary classification, plus a zero-shot TinyLlama-1.1B for three-way detection.

In practice

• Use DeBERTa-v3-large for high-performance conspiracy detection.
• Avoid heuristic relabeling of uncertain instances based on markers.

Topics

• Conspiracy Detection
• Psycholinguistics
• Transformer Models
• SemEval-2026
• DeBERTa-v3-large
• Natural Language Processing

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• mdok-style at SemEval-2026 Task 10: Finetuning LLMs for Conspiracy Detection — Data augmentation and self-training can effectively finetune LLMs for specialized text classification tasks with limited data.
• Understanding Sarcasm in Generative AI Models: How ChatGPT Interprets Human Intent — Sarcasm detection in AI requires understanding non-literal meaning through contextual learning, moving beyond simple keyword analysis.
• Lexical and Syntactic Characterization of Fake News in Portuguese Produced by Humans and by Machines — Machine-generated Portuguese fake news shows distinct lexical and syntactic patterns compared to human-produced content.
• MentalMARBERT: Domain-Adaptive Pre-training and Two-Stage Fine-Tuning for Arabic Mental Health Disorders Detection — Domain-adaptive pre-training and hierarchical classification significantly improve Arabic mental health disorder detection.
• Robust Dual-Signal Fusion: Hybrid Neuro-Symbolic Gating with Compressed Chain-of-Thought Refinement for Irony Detection in Social Media Texts — Hybrid neuro-symbolic fusion with compressed Chain-of-Thought effectively overcomes LLM literalism for zero-shot irony detection.
• A Multitask Transformer for Offensive Language Detection and Target Identification in HateBR — Multitask learning improves offensive language detection consistency but may reduce fine-grained target attribution.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.