Evaluating LLM Workflows for Generating Clinical Communication Assessment Items: A Comparative Study with Subject-Matter Experts

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Medical Specialties & Subspecialties · Depth: Advanced, quick

Summary

This study evaluated LLM-supported workflows for generating patient-centered communication assessment items in medical education. Researchers compared two content generation approaches: constrained linear and exploratory branching, both implemented with and without anchoring in vetted multiple-choice questions (MCQs). Ten subject-matter experts (SMEs) assessed 80 communication items across six quality dimensions using structured rubrics. The constrained linear approach received higher ratings, especially for medical accuracy and alignment with learning objectives and patient-centered behaviors, compared to exploratory branching. MCQ anchoring did not enhance medical accuracy. A minority of items met all criteria without revision, and no items were unanimously approved by all SMEs. These findings highlight the critical role of workflow design in LLM-supported assessment content generation, the ongoing necessity for human oversight, and the current limitations of automated content generation within medical education.

Key takeaway

For Directors of AI/ML overseeing content development in medical education, you should prioritize implementing constrained linear LLM workflows over exploratory branching for generating assessment items. Recognize that human subject-matter expert oversight remains critical, as automated content generation still requires significant revision and cannot be unanimously approved. Do not assume MCQ anchoring alone will guarantee medical accuracy; instead, focus on robust workflow design and integrated human review to ensure quality and alignment with learning objectives.

Key insights

Constrained linear LLM workflows outperform exploratory methods for medical assessment item generation, but human oversight remains crucial.

Principles

• Workflow design significantly impacts LLM-generated content quality.
• Human oversight is indispensable for complex, knowledge-rich domains.
• LLM content generation has limitations in medical education.

Method

Compared constrained linear and exploratory branching LLM workflows for generating clinical communication assessment items, with and without MCQ anchoring, evaluated by 10 SMEs across 6 quality dimensions.

In practice

• Prioritize constrained linear workflows for medical content.
• Integrate human SMEs for rigorous content review.
• Do not rely solely on MCQ anchoring for accuracy.

Topics

• LLM Workflows
• Medical Education
• Assessment Content Generation
• Clinical Communication
• Subject-Matter Experts
• Generative AI Evaluation

Best for: AI Scientist, Research Scientist, Director of AI/ML

Related on AIssential

• Generative AI Meets Data Engineering: What Happens When You Can Describe Your Database in Plain… — LLMs can generate database schemas from natural language, but require robust validation and human oversight for production reliability.
• Severity-Aware Curriculum Learning with Multi-Model Response Selection for Medical Text Generation — A severity-aware multi-model framework using curriculum learning and response selection significantly improves medical text generation across case complexities.
• The Clinical Reality Check: Why “Doctor-Chatbots” Ace Exams but Struggle in the Ward — and What Fixes It — LLMs excel in static medical QA but struggle with the iterative, information-gathering nature of real clinical diagnosis.
• Engineering a human-aligned LLM evaluation workflow with Prodigy and DSPy — Human-aligned LLM evaluation requires iterative workflows that integrate granular human feedback to engineer context-specific metrics.
• Content for Content’s Sake — LLM-generated content is subtly altering human language and communication, eroding trust and overwhelming digital systems.
• Chat2Workflow: A Benchmark for Generating Executable Visual Workflows with Natural Language — Automating executable visual workflow generation from natural language remains a significant challenge for current LLMs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.