We Tested 25 Local LLMs for Medical Use. Here’s What Shipped.

2026-06-03 · Source: Towards AI - Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Medical Devices & Health Technology · Depth: Advanced, long

Summary

Meda AI's process for selecting local LLMs for an on-premise medical AI assistant involved extensive testing of 25 models over five days, focusing on data privacy and patient safety for tasks like SOAP note generation, ICD-10-GM coding, and billing. They found that a Mac Studio M3 Ultra with 96 GB unified memory is viable for multi-model serving, achieving 110-205 tok/s with concurrency, while a Threadripper 7980X + NVIDIA 5090 (32GB VRAM) suits single-doctor deployments at 116 tok/s. Models were evaluated across three distinct stages, demanding different behaviors, using German-language transcripts. Initial screening filtered candidates by memory footprint (<25 GB), German proficiency, commercial license, and OpenAI-compatible serving. The final production stack utilizes Gemma 4 E4B for SOAP and billing, Gemma 3 4B for patient summaries, and Qwen3-30B A3B for ICD coding, consuming approximately 33 GB VRAM on an M3 Ultra.

Key takeaway

For AI Engineers building on-premise medical AI assistants, you should adopt a multi-model, multi-stage architecture to balance performance and safety. Prioritize GGUF runtimes for broader model compatibility over MLX-specific optimizations, and carefully evaluate models for specific hallucination types rather than just speed. Consider sharing model slots for similar structured extraction tasks like SOAP generation and billing to optimize memory usage on hardware like the M3 Ultra.

Key insights

Local LLMs are viable for medical AI, but require careful selection and a multi-model, multi-stage architecture.

Principles

• Multi-layer model architecture is essential for diverse clinical tasks.
• Runtime portability (e.g., GGUF) often outweighs MLX-specific optimizations.
• Hallucination types matter; grade by severity, not just rate.

Method

The process involved breaking medical AI tasks into three stages, screening 25 LLMs by hard constraints, and benchmarking survivors on hallucination rate, tokens/second, and memory footprint, then pivoting runtimes.

In practice

• Use Gemma 4 E4B GGUF for zero-hallucination SOAP extraction.
• Deploy Qwen3-30B A3B for robust ICD-10-GM coding.
• Reuse Gemma 4 for both SOAP and billing tasks.

Topics

• Local LLMs
• Medical AI
• On-premise Deployment
• LLM Benchmarking
• GGUF Runtime
• ICD-10-GM Coding

Best for: AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Class of LLMs: Benchmarking Large Language Models on the Brazilian National Medical Examination — Generalist LLMs often outperform specialized medical fine-tunes on high-stakes, language-specific medical exams.
• Your AI Doesn’t Know Anything. And That’s Not the Model’s Fault. — AI hallucinations stem from infrastructure limitations, not model intelligence, necessitating robust knowledge layers like graphs.
• [AINews] Top Local Models List - April 2026 — Community consensus on local LLMs prioritizes practical usability and specific use cases over raw benchmark supremacy.
• Ollama: Unleash the Power of Local LLMs on Your Machine — Ollama enables private, cost-effective, and low-latency local LLM inference and customization via a Python library.
• LLM Building Blocks & Transformer Alternatives — Advanced transformer architectures and emerging alternatives aim to balance model scale, inference efficiency, and specialized capabilities.
• GPT-5.5 Outperforms (and Hallucinates), Kimi K2.6 Leads Open LLMs, AI Strains Climate Pledges, Strategic Thinking in LLMs vs. Humans — Advanced LLMs like GPT-5.5 and Kimi K2.6 push performance boundaries but reveal diverging objective and subjective evaluation outcomes.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards AI - Medium.