Vasudev Awatramani at #SMM4H-HeaRD 2026: A Two-Pass LLM Pipeline with Deterministic Rule Derivation for Interpretable Insomnia Detection in Clinical Notes

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Science & Research — Artificial Intelligence & Machine Learning, Health & Medical Research, Natural Language Processing · Depth: Expert, quick

Summary

Vasudev Awatramani's system for the #SMM4H-HeaRD 2026 Shared Task 2 addresses insomnia detection within MIMIC-III clinical notes using a two-pass large language model pipeline. The first pass employs a Gemini 2.5 Flash LLM, prompted via BAML, to extract structured evidence like sleep difficulties, daytime impairment, and hypnotic medications, complete with verbatim character-level citations. Subsequently, a small Python rule engine deterministically applies published Insomnia rules—Definition 1, Definition 2, and Rules B and C—to generate binary patient-level labels, rule-component labels, and their corresponding evidence spans. The zero-shot variant of this system achieved an F1 score of 0.8108 on Subtask 1 (binary classification) and a micro-F1 of 0.7126 with partial-match span F1 of 0.6621 on Subtask 2, both surpassing the across-team mean. A retrieval-augmented few-shot variant improved Subtask 1 precision but not overall F1.

Key takeaway

For NLP Engineers developing clinical decision support systems, if you are prioritizing interpretability and auditability in sensitive health applications, consider a two-pass LLM architecture. Instead of directly asking the LLM for a diagnosis, use it to extract structured evidence from notes, then apply a deterministic rule engine. This approach, demonstrated with Gemini 2.5 Flash, yields strong, auditable results for tasks like insomnia detection, offering greater transparency than end-to-end LLM label prediction.

Key insights

Combining LLM evidence extraction with deterministic rule application provides interpretable clinical label prediction.

Principles

• Evidence extraction + rule application enhances interpretability.
• Deterministic rule engines consuming LLM output yield auditable behavior.
• Few-shot learning may improve precision without improving F1.

Method

A two-pass pipeline where a Gemini 2.5 Flash LLM extracts structured evidence from clinical notes via BAML prompts, followed by a Python rule engine applying predefined insomnia rules to derive labels and evidence spans.

In practice

• Use BAML for typed LLM prompts to structure evidence.
• Combine LLMs with rule engines for auditable clinical NLP.
• Evaluate zero-shot vs. few-shot carefully for F1.

Topics

• LLM Pipeline
• Insomnia Detection
• Clinical NLP
• MIMIC-III
• Evidence Extraction
• Rule-Based Systems
• Interpretable AI

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• T2D-Bench: Evidence-Gated Evaluation of LLM Outputs for Type 2 Diabetes Using a Multi-Layer Clinical-Lifestyle Knowledge Graph — LLMs require evidence-gated evaluation against clinical knowledge graphs to ensure medical accuracy and guideline compliance.
• SymptomWise: A Deterministic Reasoning Layer for Reliable and Efficient AI Systems — SymptomWise improves AI diagnostic reliability by separating language understanding from deterministic, knowledge-driven reasoning.
• Reading between the Lines: Leveraging Large Language Models for Global Dementia and Depression Assessment from Clinical Interviews — LLMs can effectively assess neuropsychiatric disorder severity from speech, enabling automated screening.
• Clinically Interpretable Sepsis Early Warning via LLM-Guided Simulation of Temporal Physiological Dynamics — LLM-guided temporal simulation improves sepsis early warning by providing interpretable physiological trajectory predictions.
• LLM-Orchestrated Conformance Checking in Stroke Care Without Computer-Interpretable Guidelines — LLM orchestration enables healthcare conformance checking directly from unstructured text, bypassing CIGs.
• Language Models as Interfaces, Not Oracles: A Hybrid LLM-ML System for Pediatric Appendicitis — Hybrid AI systems improve clinical decision support by using LLMs for feature extraction and validated ML models for reliable, auditable prediction.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.