Automated Detection and Classification of Delusion-related Content in Naturalistic Audio Diaries Using Multi-Agent Language Models

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

A novel automated, multi-agent Large Language Model (LLM) pipeline has been developed for the fine-grained, multi-label extraction of language suggesting delusional beliefs, associated affective responses, and behavioral responses. This pipeline processes transcripts from naturalistic audio diaries collected from individuals with moderate persecutory ideation. The research evaluated an ensemble of three foundation models, demonstrating that detailed diagnostic prompt instructions effectively reduce false positives for delusional theme classification, though they also constrain the interpretation of affective or behavioral responses. Critically, comparing multi-agent adjudication frameworks revealed that complex conversational debate between agents decreased accuracy on clinically ambiguous text by inducing premature consensus. Instead, a majority voting approach established robust performance, achieving a Micro F1 score of 0.872 for delusion detection and 0.779 for classification. This work offers a validated and scalable method for characterizing delusion-related content in naturalistic speech.

Key takeaway

For NLP Engineers developing clinical text analysis systems, this research suggests prioritizing simpler adjudication methods over complex conversational agents for ambiguous data. If you are building LLM-based tools for mental health applications, you should implement detailed diagnostic prompts to reduce false positives, but be aware they may limit the interpretation of nuanced responses. Consider majority voting for robust classification of sensitive clinical text.

Key insights

The multi-agent LLM pipeline effectively detects delusion-related content in naturalistic speech transcripts using majority voting.

Principles

• Detailed diagnostic prompts reduce false positives.
• Conversational debate can diminish accuracy on ambiguous text.
• Majority voting ensures robust performance for clinical text.

Method

A multi-agent LLM pipeline extracts delusion-related language, affect, and behavior from audio diary transcripts. It uses an ensemble of foundation models with diagnostic prompts and adjudicates classifications via majority voting.

In practice

• Automate mental illness symptom detection.
• Characterize delusion phenomenology scalably.
• Evaluate LLMs for clinical text analysis.

Topics

• Multi-Agent LLMs
• Delusion Detection
• Clinical NLP
• Audio Diary Analysis
• Mental Health AI
• Natural Language Processing

Best for: AI Scientist, Research Scientist, NLP Engineer

Related on AIssential

• An Agentic LLM-Based Framework for Population-Scale Mental Health Screening — An agentic LLM framework enhances mental health screening by ensuring robust, adaptable, and trustworthy AI pipelines.
• Listening Between the Lines: Joint Learning of ASR Embeddings and LLM-Augmented Linguistics for Dementia Detection — Jointly learning ASR embeddings and LLM-augmented linguistics significantly improves dementia detection from speech.
• Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations — NLAs translate LLM activations into human-readable text, revealing internal thoughts and hidden motivations for auditing.
• Can AI Chatbots Reason Like Doctors? — Large language models demonstrate potential in clinical reasoning, but require careful integration and standardized evaluation in medical workflows.
• GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations — LLM agent memory systems perform poorly in multi-party conversations, often outperformed by simple baselines.
• Multi-View Attention Multiple-Instance Learning Enhanced by LLM Reasoning for Cognitive Distortion Detection — Combining LLMs with MIL and ELB components improves cognitive distortion detection by enhancing interpretability and reasoning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.