Human-in-the-Loop Large Language Model Framework for Identification of Cutaneous Immune-Related Adverse Events

2026-07-25 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Medical Devices & Health Technology · Depth: Expert, quick

Summary

The study evaluated a retrieval-augmented, multi-agent large language model (LLM)-driven, human-in-the-loop framework designed for detecting cutaneous immune-related adverse events (cirAEs) from clinical notes. This LLM-assisted workflow demonstrated significant improvements over unassisted manual review. Specifically, it achieved an F1 score of 0.88 compared to 0.77 for manual review, and inter-rater agreement, measured by Cohen's kappa, increased from 0.50 to 0.82. Furthermore, the framework reduced the average review time by approximately half. This research pilots a method for applying LLMs to identify immune-related toxicities across various organ systems, aiming to enable accurate, scalable, and transparent adverse event data extraction.

Key takeaway

For clinical researchers and pharmacovigilance teams analyzing adverse event data, this human-in-the-loop LLM framework offers a path to significantly improve both the accuracy and efficiency of identifying cutaneous immune-related adverse events. You should consider piloting similar retrieval-augmented, multi-agent LLM systems to reduce review times by half and enhance inter-rater agreement, thereby scaling your adverse event detection capabilities across various organ systems.

Key insights

A human-in-the-loop LLM framework significantly improves accuracy and efficiency in identifying cutaneous immune-related adverse events.

Principles

• Human-in-the-loop LLM frameworks enhance clinical data extraction.
• LLMs can scale adverse event identification across organ systems.

Method

The framework utilizes a retrieval-augmented, multi-agent large language model with human oversight to process clinical notes for detecting cutaneous immune-related adverse events.

In practice

• Implement LLMs for adverse event data extraction.
• Combine LLMs with human review for high-stakes tasks.

Topics

• Large Language Models
• Human-in-the-Loop AI
• Adverse Event Detection
• Pharmacovigilance
• Clinical Notes Analysis
• Immune-Related Adverse Events

Best for: NLP Engineer, AI Scientist, Research Scientist, Domain Expert

Related on AIssential

• Cross-Contextual Vision-Language Adaptation with LoRA for Personalized Severe Adverse Event Detection in Clinical Wound Monitoring — Cross-contextual LoRA fusion and temporal OOD detection enhance personalized severe adverse event identification in wound monitoring.
• AIPatient Arena: EHR-grounded evaluation of large language models in end-to-end clinical consultation workflows — AIPatient Arena reveals LLMs need process-based evaluation beyond final answers for clinical readiness, highlighting interaction failures.
• WHBench: Evaluating Frontier LLMs with Expert-in-the-Loop Validation on Women's Health Topics — LLMs significantly underperform on women's health clinical scenarios, particularly in safety and health equity, despite strong general medical benchmark results.
• AI scans 400,000 Reddit posts and finds hidden Ozempic side effects — AI-driven social media analysis can rapidly identify underreported drug side effects missed by traditional clinical trials.
• LLM-Orchestrated Conformance Checking in Stroke Care Without Computer-Interpretable Guidelines — LLM orchestration enables healthcare conformance checking directly from unstructured text, bypassing CIGs.
• Please don’t trust your chatbot for medical advice — LLMs are unreliable for medical advice, frequently generating misinformation with overconfidence and lacking clinical reasoning.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.