Bhramastra at #SMM4H-HeaRD 2026: A Multi-Stage Hunter-Judge Pipeline using DSPy-Optimized LLMs for Multilingual ADE Detection

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Medical Natural Language Processing · Depth: Expert, quick

Summary

Team Bhramastra's submission for the #SMM4H-HeaRD 2026 Shared Task 1 addresses personal Adverse Drug Event (ADE) detection in multilingual social media. The team proposed a decoupled "Hunter-Judge" architecture to manage extreme class imbalance and linguistic diversity across seven languages, including a surprise zero-shot Farsi set. This system utilizes a fine-tuned multilingual mDeBERTa-v3 model as a high-recall "Hunter" filter. Subsequently, a Gemini-2.5-Flash model, optimized via the DSPy framework, functions as a "Judge" for precision-oriented agentic adjudication. By integrating a reasoning protocol based on clinical RAG evidence and universal ingredient mapping, the pipeline achieved the highest average F1-score of 0.6653 among all participating teams. It also demonstrated strong zero-shot generalizability on Farsi, with an F1-score of 0.5863.

Key takeaway

For NLP Engineers developing multilingual health monitoring systems, consider adopting a decoupled Hunter-Judge architecture. This approach, combining a high-recall filter like mDeBERTa-v3 with a DSPy-optimized LLM (e.g., Gemini-2.5-Flash) for precision, effectively addresses class imbalance and linguistic variance. Integrating clinical RAG and universal ingredient mapping can significantly boost zero-shot generalization, especially in low-resource languages, improving overall ADE detection accuracy.

Key insights

A decoupled Hunter-Judge architecture with DSPy-optimized LLMs effectively detects multilingual ADEs, achieving top performance and zero-shot generalization.

Principles

• Decoupled architecture handles class imbalance and linguistic variance.
• Medically-grounded reasoning improves low-resource generalization.
• DSPy optimizes LLMs for precision-oriented agentic tasks.

Method

A multi-stage pipeline uses mDeBERTa-v3 as a high-recall "Hunter" filter, followed by a DSPy-optimized Gemini-2.5-Flash "Judge" for precision-oriented adjudication, leveraging clinical RAG and ingredient mapping.

In practice

• Combine fine-tuned models with LLMs for complex tasks.
• Use DSPy for LLM optimization in specific roles.
• Integrate RAG for evidence-based reasoning in LLM pipelines.

Topics

• Adverse Drug Event Detection
• Multilingual NLP
• LLM Optimization
• DSPy
• Hunter-Judge Architecture
• Social Media Monitoring
• Gemini-2.5-Flash

Best for: AI Engineer, Machine Learning Engineer, AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• MADE: Beyond Scoring via a Multilingual Agentic Diagnosing Engine for Fine-Grained Evaluation Insights — MADE offers a structured, agentic approach for fine-grained, multilingual LLM evaluation diagnosis, moving beyond simple benchmark scores.
• Multi-View Decompilation for LLM-Based Malware Classification — Using multiple decompiler views improves LLM-based malware classification by providing complementary evidence.
• Analysing Lightweight Large Language Models for Biomedical Named Entity Recognition on Diverse Ouput Formats — Lightweight LLMs offer competitive biomedical NER performance, with specific output formats proving more effective than multi-format tuning.
• Detoxification for LLM: From Dataset Itself — Detoxifying LLM pretraining datasets directly at the source fundamentally reduces inherent model toxicity while preserving semantics.
• Prompt, Plan, Extract: Zero-Shot Agentic LLMs Workflows for Lung Pathology Extraction from Clinical Narratives — Zero-shot agentic LLMs can accurately extract complex lung pathology data from clinical narratives without specific training.
• Multilingual Detection of Alzheimer's Disease from Speech: A Cross-Linguistic Transfer Learning Approach — Cross-language transfer learning effectively detects Alzheimer's Disease from speech across multiple languages, achieving strong generalization and rapid inference.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.