MHGraphBench: Knowledge Graph-Grounded Benchmarking of Mental Health Knowledge in Large Language Models

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, AI in Mental Health · Depth: Expert, quick

Summary

MHGraphBench introduces a knowledge-graph (KG)-grounded benchmark designed to evaluate large language models (LLMs) on their mental health knowledge and application to clinically salient structured judgments. Derived from PrimeKG, this benchmark features nine task families, including entity recognition, relation judgment, and two-hop reasoning, all supported by KG-derived answers and controlled negative options. Experiments across 15 closed- and open-source LLMs revealed a consistent recognition-to-judgment gap; while leading models performed well on entity typing, they struggled significantly with relation prediction and two-hop reasoning. The study also found that short KG-derived snippets had varied effects, sometimes degrading performance, and that output-format reliability critically influenced measured performance in constrained multiple-choice settings. MHGraphBench specifically evaluates agreement with a curated mental-health slice of PrimeKG under a constrained multiple-choice interface, rather than real-world clinical safety.

Key takeaway

For machine learning engineers developing LLMs for mental health applications, you must prioritize robust evaluation beyond basic entity recognition. The MHGraphBench findings highlight a significant recognition-to-judgment gap, indicating that models struggle with complex relation prediction and two-hop reasoning despite strong entity typing. Therefore, focus your development and fine-tuning efforts on improving these advanced reasoning capabilities, and carefully consider how benchmark design and output format reliability influence your model's measured performance.

Key insights

LLMs exhibit a recognition-to-judgment gap in mental health knowledge, struggling with complex reasoning despite strong entity recognition.

Principles

• LLM mental health knowledge is inconsistent.
• Complex reasoning tasks challenge LLMs more.
• Benchmark design impacts measured performance.

Method

MHGraphBench assesses LLMs using a PrimeKG-derived benchmark with nine task families for entity recognition, relation judgment, and two-hop reasoning, employing KG-supported answers and controlled negative options.

In practice

• Evaluate LLMs on mental health entity recognition.
• Test LLM relation prediction capabilities.
• Assess two-hop reasoning in clinical contexts.

Topics

• Large Language Models
• Mental Health
• Knowledge Graphs
• LLM Benchmarking
• Clinical Reasoning
• PrimeKG

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• GTBench: A Curriculum-Grounded Benchmark for Evaluating LLMs as Mathematical Research Assistants in Graph Theory — LLMs struggle with mathematical reasoning, particularly graduate-level proofs, with GPT-5 significantly outperforming other models.
• Team MKC at CLPsych 2026: Capturing and Characterizing Mental Health Changes through Social Media Timeline Dynamics — An LLM-based pipeline analyzes social media timelines for early mental health change detection and continuous monitoring, integrating post and user-level assessments.
• ActTraitBench: Quantifying the Knowledge-Decision Gap in Large Language Models via Human-Grounded Behavioral Validation — ActTraitBench quantifies the Knowledge-Decision Gap in LLMs, revealing behavioral divergence despite consistent self-reports.
• AttuneBench: A Conversation-Based Benchmark for LLM Emotional Intelligence — LLM emotional intelligence is a multi-faceted capability, requiring distinct evaluation metrics beyond simple emotion recognition.
• Creating Multilingual Mental Health Dialogue Datasets: Limits of Persona-Based Localization via Nationality and Language — Simple nationality and language parameter changes in English-centric personas fail to preserve clinical consistency in multilingual mental health dialogues.
• Active Evidence-Seeking and Diagnostic Reasoning in Large Language Models for Clinical Decision Support — LLMs exhibit a substantial performance gap in active, multi-turn clinical diagnosis compared to static, full-context evaluations.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.