AlienAnnotators at PsyDefDetect: What Lies Between the Lines: Probing Lightweight Open-Source LLMs for Psychological Defense Mechanism Detection

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Science & Research — Artificial Intelligence & Machine Learning, Health & Medical Research · Depth: Advanced, quick

Summary

AlienAnnotators at PsyDefDetect conducted a systematic analysis for the BioNLP@ACL 2026 shared task, focusing on detecting psychological defense mechanisms in therapy dialogue. This task involves a nine-class utterance-level classification based on the Defense Mechanism Rating Scale (DMRS). Researchers evaluated six open-source, instruction-tuned small language models (SLMs, <= 9B parameters) in both zero-shot and fine-tuning configurations, comparing a clinically-grounded prompt against a baseline. Their official submission achieved 59.96% accuracy and 16.28% Macro F1. Post-submission experiments, incorporating fine-tuning with 5-fold cross-validation and logit averaging, significantly boosted performance to 65.25% accuracy and 34.59% Macro F1. Findings indicate clinically-grounded prompts are superior, model scale doesn't guarantee zero-shot improvement, and fine-tuning is crucial for performance recovery, though some defense tiers remain challenging due to clinical ambiguity.

Key takeaway

For NLP Engineers developing systems for psychological defense mechanism detection, prioritize fine-tuning lightweight open-source LLMs. Your models, even those under 9B parameters, can achieve significantly higher accuracy and Macro F1 scores (up to 65.25% and 34.59% respectively) when combined with clinically-grounded prompts and ensemble methods like logit averaging. Focus on developing domain-specific prompts and robust fine-tuning strategies to overcome zero-shot limitations and address inherent clinical ambiguities in defense tier boundaries.

Key insights

Fine-tuning lightweight LLMs with clinically-grounded prompts significantly improves psychological defense mechanism detection, overcoming zero-shot limitations.

Principles

• Clinically-grounded prompts outperform bare definitions.
• Fine-tuning dramatically recovers collapsed zero-shot models.
• Clinical ambiguity limits defense tier detection.

Method

Systematic evaluation of six SLMs (<= 9B parameters) using zero-shot and fine-tuning. Employed 5-fold cross-validation and logit averaging ensemble for improved classification of nine defense mechanisms.

In practice

• Use DMRS for nine-class utterance classification.
• Apply fine-tuning with cross-validation for SLMs.
• Develop clinically-grounded prompts for better results.

Topics

• Psychological Defense Mechanisms
• Small Language Models
• Fine-tuning LLMs
• Utterance Classification
• Clinically-grounded Prompts
• BioNLP@ACL 2026

Best for: NLP Engineer, AI Scientist, Research Scientist

Related on AIssential

• LinguIUTics at PsyDefDetect: Iterative Imbalance-Aware Fine-tuning of Qwen3-8B for Psychological Defense Mechanism Classification — Iterative imbalance-aware fine-tuning of LLMs, combined with augmentation and post-processing, significantly improves rare-class NLP performance.
• Improving Alignment Between Human and Machine Codes: An Empirical Assessment of Prompt Engineering for Construct Identification in Psychology — Empirical prompt engineering, focusing on construct definition and examples, significantly improves LLM text classification alignment with human expert judgment.
• Reading between the Lines: Leveraging Large Language Models for Global Dementia and Depression Assessment from Clinical Interviews — LLMs can effectively assess neuropsychiatric disorder severity from speech, enabling automated screening.
• ttda704 at SemEval-2026 Task 6: Structured Chain-of-Thought Prompting for Political Evasion Detection — Structured Chain-of-Thought prompting significantly improves political evasion detection over fine-tuning large language models.
• Modeling Pathology-Like Behavioral Patterns in Language Models Through Behavioral Fine-Tuning — Behavioral fine-tuning can induce specific, stable, pathology-like biases in LLMs, shifting their fundamental interpretive priors.
• DeEscalWild: A Real-World Benchmark for Automated De-Escalation Training with SLMs — Domain-specific fine-tuning of SLMs on real-world data surpasses larger generalist LLMs for high-stakes de-escalation training.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.