LinguIUTics at PsyDefDetect: Iterative Imbalance-Aware Fine-tuning of Qwen3-8B for Psychological Defense Mechanism Classification

2026-05-30 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

LinguIUTics achieved a macro F1-score of 0.3917 in the PsyDefDetect 2026 shared task for psychological defense mechanism classification, ranking 4th among 21 teams. This represents a 7.7 absolute point (24.4% relative) improvement over the Ministral-8B baseline of 31.48 macro F1. Addressing severe class imbalance that hindered BERT-family encoders and zero-shot LLMs on rare classes, the team employed QLoRA fine-tuning of Qwen3-8B. Their methodology incorporated grouped stratified cross-validation to prevent data leakage, minority-class round-robin lexical augmentation, and a post-processing pipeline featuring logit bias tuning and ensemble blending. These strategies significantly enhanced minority-class recall, boosting the critical "Unclear" class (Level 8) from near-zero performance to an F1 score of 0.797.

Key takeaway

For NLP Engineers developing classification systems with severe class imbalance, particularly in clinical contexts like psychological defense mechanisms, you should prioritize QLoRA fine-tuning of models like Qwen3-8B. Implement grouped stratified cross-validation and minority-class lexical augmentation to improve rare-class recall. Your post-processing pipeline should include logit bias tuning and ensemble blending to achieve substantial performance gains on critical, underrepresented categories.

Key insights

Iterative imbalance-aware fine-tuning of LLMs, combined with augmentation and post-processing, significantly improves rare-class NLP performance.

Principles

• Class imbalance severely degrades rare-class performance for BERT-family and zero-shot LLMs.
• Grouped stratified cross-validation prevents data leakage in imbalanced datasets.
• Combining augmentation, fine-tuning, and post-processing boosts minority-class recall.

Method

QLoRA fine-tuning of Qwen3-8B, grouped stratified cross-validation, minority-class round-robin lexical augmentation, and a post-processing pipeline with logit bias tuning and ensemble blending.

In practice

• Apply QLoRA fine-tuning to LLMs for imbalanced text classification.
• Use grouped stratified cross-validation for robust model evaluation.
• Implement lexical augmentation for underrepresented classes.

Topics

• Psychological Defense Mechanisms
• Utterance Classification
• Class Imbalance
• QLoRA Fine-tuning
• Qwen3-8B
• Lexical Augmentation
• Ensemble Blending

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Companies Are Using Reddit to Manipulate ChatGPT and Google AI Search — Companies are manipulating AI search results and LLM outputs by spamming specific online communities like Reddit.
• Article: Why Vector Search Alone Isn't Enough: Hybrid Retrieval for RAG — Hybrid retrieval combining vector search and BM25 via Reciprocal Rank Fusion is essential for robust RAG systems handling diverse query types.
• Google unveils its intelligent search box, Sonata posts growth from AI services as Fortune 500 reaches an all-time high: May Roundup — The AI industry is rapidly maturing, moving from tentative adoption to enterprise-wide integration and specialized applications.
• TalkTag: Fine-Grained Morphosyntactic Error Annotation for Transcribed Speech — TalkTag automates fine-grained morphosyntactic error annotation in spoken transcripts using an LLM, even with scarce data.
• EvoPool: Evolutionary Programmatic Annotation for Label-Efficient Specialized Supervision — EvoPool uses evolutionary multi-agent programmatic annotation to create label-efficient, specialized supervision systems that outperform LLMs in high-stakes domains.
• An NLP-Driven Framework for Curriculum-Labor Market Alignment: Schema-Constrained LLM Extraction, ESCO-Anchored Semantic Matching, and Multi-Dimensional Gap Quantification — An NLP framework uses LLMs and semantic matching to quantify curriculum-labor market competency gaps with high reliability.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.