Measuring the practice of shared-decision making (OPTION12): An Investigation into Open-sourced Smaller LLMs (OS-sLLMs) for Better Privacy and Sustainability

2026-07-08 · Source: cs.CL updates on arXiv.org · Field: Health & Wellbeing — Clinical Care & Medical Practice, Medical Devices & Health Technology, Health & Medical Research · Depth: Expert, extended

Summary

This study investigates the capability of open-source smaller Large Language Models (OS-sLLMs) to automate the coding of Shared Decision-Making (SDM) in clinical consultations using the 12-item OPTION12 instrument. Researchers analyzed 26 Dutch melanoma patient consultation transcripts, double-coded by human experts. The methodology involved a pilot study for prompt refinement and sLLM selection, followed by deployment on a testing set, incorporating a "Judge-sLLM" to resolve disagreements. Five OS-sLLMs were tested, including general domain models (Gemma3:12b, Llama3.1:8b, Mistral7b) and medical domain models (Meditron, MedLlama). Preliminary results indicate general domain models consistently outperformed medical ones, which frequently hallucinated. Mistral7b showed 4 consensus items with human coding, while Gemma3:12b and Llama3.1:8b achieved 3. Pearson correlations with human coding were 0.83 (Gemma3:12b), 0.80 (Llama3.1:8b), and 0.64 (Mistral7b) on the testing set. The study identifies specific error categories and suggests OS-sLLMs hold promise as supportive tools within human-in-the-loop annotation pipelines for privacy and sustainability.

Key takeaway

For AI Scientists and Machine Learning Engineers developing clinical annotation tools, you should prioritize general-domain open-source smaller LLMs over specialized medical models for discourse analysis tasks like Shared Decision-Making assessment. Your focus should be on robust prompt engineering, potentially incorporating a "Judge-LLM" framework to resolve model disagreements, and designing human-in-the-loop systems. This approach can enhance privacy and sustainability while accelerating annotation processes, even if full automation is not yet feasible.

Key insights

Open-source smaller LLMs show promise for automating shared decision-making assessment, particularly general domain models.

Principles

• General LLMs excel over specialized medical LLMs for discourse analysis tasks.
• OPTION12 coding relies on conversational reasoning, not primarily medical knowledge.
• Model augmentation can improve performance across diverse assessment items.

Method

The proposed "Judge-LLM consensus framework" mimics human annotator disagreement resolution. It involves prompt refinement, sLLM selection, few-shot learning, and a judge-sLLM to reconcile scores from other OS-sLLMs.

In practice

• Deploy OS-sLLMs locally for privacy-preserving clinical data analysis.
• Integrate LLMs into human-in-the-loop workflows to reduce annotation burden.
• Focus prompt engineering on temporal reasoning and role attribution for SDM tasks.

Topics

• Shared Decision Making
• OPTION12 Instrument
• Open-source Smaller LLMs
• Clinical Consultation Analysis
• Privacy-Preserving AI
• Human-in-the-Loop

Best for: NLP Engineer, AI Scientist, Research Scientist, Machine Learning Engineer

Related on AIssential

• SLMJury: Can Small Language Models Judge as Well as Large Ones? — Small Language Models can effectively judge model outputs, but their performance is highly dependent on the task domain and judging paradigm.
• CLEAR: Revealing How Noise and Ambiguity Degrade Reliability in LLMs for Medicine — LLMs exhibit a "humility deficit" in medical contexts, struggling with ambiguity and preferring incorrect answers over admitting uncertainty.
• Open source llm (glm 4.7) matching closed models on coding benchmarks. Tested via api on real projects. — Open-source models are achieving competitive, specialized performance in domains like coding at significantly lower costs.
• Portuguese Sentiment Analysis with Open-Source LLMs: Models, Prompts, and Efficient Deployment — Open-source LLMs offer competitive Portuguese sentiment analysis with efficient deployment, especially using CoT+FS prompting.
• OLLM: Options-based Large Language Models — OLLM uses a discrete latent space to model multiple next-token options, improving LLM control and performance.
• From Fallback to Frontline: When Can LLMs be Superior Annotators of Human Perspectives? — LLMs can statistically outperform human annotators in estimating aggregate subgroup opinions under common conditions.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.