PerceptionLab at PsyDefDetect: Overcoming Extreme Response Bias in LLMs via Rubric-Grounded Retrieval and Supervised Clinical Reasoning Distillation for Fine-Grained Ordinal Classification

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mental Health & Psychological Support · Depth: Expert, short

Summary

PerceptionLab's third-place system for the PsyDefDetect Shared Task at BioNLP 2026 addresses the challenge of automating fine-grained ordinal classification of psychological defense mechanisms using Large Language Models. General-purpose LLMs typically exhibit severe extreme response bias, failing to resolve nuanced, mid-level defenses within frameworks like the Defense Mechanism Rating Scales and instead gravitating towards scale endpoints. The proposed hybrid architecture combines label-flattened generative retrieval with an LLM classifier, which is fine-tuned through the distillation of supervised clinical reasoning traces. This dual approach grounds decisions in rubric criteria and integrates task-specific supervision, successfully mitigating the observed bias. The system achieved an accuracy of 67.37% and a macro-F1 of 39.56%, demonstrating that targeted clinical supervision and dynamic rubric-grounded retrieval significantly outperform raw parameter scale foundation models.

Key takeaway

For NLP Engineers developing systems for fine-grained clinical text classification, particularly involving ordinal scales like psychological defense mechanisms, you should integrate rubric-grounded retrieval and supervised clinical reasoning distillation into your LLM pipelines. This approach directly mitigates extreme response bias, which causes models to miss nuanced, mid-level distinctions. Implementing this hybrid architecture can significantly improve accuracy and macro-F1 scores, moving beyond the limitations of un-tuned foundation models in sensitive clinical applications.

Key insights

LLMs' extreme response bias in clinical NLP can be overcome by combining rubric-grounded retrieval with supervised reasoning distillation.

Principles

• LLMs exhibit extreme response bias in ordinal clinical classification.
• Integrating rubric criteria improves fine-grained classification.
• Supervised clinical reasoning distillation enhances LLM performance.

Method

A hybrid architecture combines label-flattened generative retrieval with an LLM classifier fine-tuned by distilling supervised clinical reasoning traces, grounding decisions in rubric criteria and task-specific supervision.

In practice

• Apply label-flattened generative retrieval for context.
• Distill supervised clinical reasoning traces into LLMs.
• Use rubric criteria to ground LLM classification decisions.

Topics

• Clinical NLP
• Psychological Defense Mechanisms
• Large Language Models
• Extreme Response Bias
• Rubric-Grounded Retrieval
• Supervised Distillation

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• Retrieval-Augmented Generation for Clinical Question Answering in Portuguese Drug Leaflets: Benefits and Limitations — RAG improves factual recall in clinical Q&A but needs adaptive control for complex reasoning to avoid anchoring bias.
• Multi-View Attention Multiple-Instance Learning Enhanced by LLM Reasoning for Cognitive Distortion Detection — Combining LLMs with MIL and ELB components improves cognitive distortion detection by enhancing interpretability and reasoning.
• Measuring the sensitivity of LLM-based structured extraction to prompt, model, and schema choices in clinical discharge summaries — LLM-based clinical data extraction sensitivity varies significantly with model and schema, especially regarding "absence-versus-silence" distinctions.
• Possible or Definite? A Benchmark for Evaluating Diagnostic Uncertainty Preservation in Clinical Text — LLMs systematically distort diagnostic uncertainty in clinical text, often converting nuanced statements into definite assertions.
• Comparing LLM and Fine-Tuned Model Performance on NVDRS Circumstance Extraction with Varying Prompt Complexity — LLMs, guided by a context-aware "Complexity Score" for prompt selection, significantly outperform fine-tuned models on inferentially complex, low-prevalence data extraction.
• AI Rater Discrimination Depends on Scoring Protocol in Complex Clinical Decision-Making — Rubric-anchored scoring protocols are essential for preserving discriminative power in clinical AI evaluation, especially for patient-specific criteria.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.