Measuring the practice of shared-decision making (OPTION12): An Investigation into Open-sourced Smaller LLMs (OS-sLLMs) for Better Privacy and Sustainability

2026-07-07 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, quick

Summary

The LLM4SDM study introduces the first investigation into open-source smaller language models (OS-sLLMs) for automated assessment of shared decision making (SDM) using the Observer OPTION12 framework. Diverging from prior research that utilized large commercial models and the shorter OPTION5 instrument, this work focuses on privacy-preserving, locally deployable models applied to Dutch melanoma consultation transcripts. A pilot study evaluated three general-domain and two medical-domain OS-sLLMs, revealing that general-domain models surpassed medical-domain models, which frequently exhibited hallucination and instruction-following failures. Specifically, Gemma3:12b achieved the highest agreement with human annotations (Pearson r=0.51, Spearman \r{ho}=0.59). The research also identified systematic challenges in temporal discourse reasoning, conversational role attribution, and evidence grounding, and proposed a Judge-LLM consensus framework to aid disagreement resolution among models. While not yet human replacements, OS-sLLMs show promise for human-in-the-loop SDM assessment with enhanced privacy.

Key takeaway

For NLP Engineers developing automated clinical decision support, consider open-source smaller LLMs like Gemma3:12b for privacy-preserving shared decision making assessment. While these models achieve strong agreement with human annotations (Pearson r=0.51), your implementation must account for current limitations in temporal reasoning and conversational role attribution. Integrate a human-in-the-loop validation process and explore Judge-LLM consensus frameworks to enhance reliability and address model hallucinations.

Key insights

Open-source smaller LLMs offer a privacy-preserving foundation for automated shared decision making assessment, despite current limitations.

Principles

• General-domain OS-sLLMs can outperform medical-domain models for specific clinical NLP tasks.
• Local deployment of OS-sLLMs enables privacy-preserving analysis of sensitive clinical data.

Method

Evaluate OS-sLLMs on expert-annotated clinical consultations using OPTION12, then apply a Judge-LLM consensus framework to resolve inter-model disagreements for robust SDM assessment.

In practice

• Consider Gemma3:12b for initial automated SDM assessment in privacy-sensitive contexts.
• Implement a multi-model Judge-LLM consensus system to enhance reliability of OS-sLLM outputs.

Topics

• Shared Decision Making
• Open-source LLMs
• OPTION12 Framework
• Clinical NLP
• Privacy-preserving AI
• Gemma3:12b

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Small, Private Language Models as Teammates for Educational Assessment Design — SLMs offer competitive, privacy-sensitive assessment design, but require human oversight due to model evaluation biases.
• OLLM: Options-based Large Language Models — OLLM uses a discrete latent space to model multiple next-token options, improving LLM control and performance.
• Portuguese Sentiment Analysis with Open-Source LLMs: Models, Prompts, and Efficient Deployment — Open-source LLMs offer competitive Portuguese sentiment analysis with efficient deployment, especially using CoT+FS prompting.
• NICE: A Theory-Grounded Diagnostic Benchmark for Social Intelligence of LLMs — NICE provides a theory-grounded diagnostic benchmark to identify specific social intelligence weaknesses in LLMs, crucial for their application in social contexts.
• Teaching LLMs Brazilian Healthcare: Injecting Knowledge from Official Clinical Guidelines — Domain-specific synthetic data and reinforcement learning significantly enhance LLM performance on specialized clinical knowledge.
• The Architecture Behind Open-Source LLMs — MoE architectures and diverse training strategies drive rapid progress in open-weight LLMs, optimizing cost and performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.