ACSS-PSL at #SMM4H-HeaRD 2026: An LLM-Driven Autoresearch Loop for Opioid-Impact NER

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, short

Summary

ACSS-PSL presented an LLM-driven autoresearch protocol for Task 7 of #SMM4H-HeaRD 2026, focusing on extracting ClinicalImpacts and SocialImpacts from Reddit posts concerning non-medical opioid use. This protocol involved a coding agent that iteratively proposed hypotheses, modified training configurations, and evaluated against a held-out validation set. The study found a narrow viable search space on the small dataset of 842 training examples, with only 9 out of 79 runs improving strict F1 scores. The final submitted ensemble combined DeBERTa-large, MC Dropout blending, and a constrained multi-LLM consensus layer, achieving 0.46 strict F1 and 0.52 relaxed F1 on the test set. The authors note that single-seed evaluation limits the reliability of run-level comparisons, but the comprehensive run log offers a reproducible case study of autonomous experimentation, detailing failure modes and guardrails for small-data Named Entity Recognition.

Key takeaway

For NLP Engineers or Research Scientists optimizing Named Entity Recognition models on sensitive, small datasets, consider implementing an LLM-driven autoresearch loop to systematically explore model configurations. Be prepared for a narrow viable search space, as demonstrated by only 9 out of 79 runs improving strict F1 on 842 training examples. Prioritize robust ensemble techniques, such as combining DeBERTa-large with MC Dropout blending and a multi-LLM consensus layer, to achieve reliable performance and mitigate risks associated with limited data.

Key insights

An LLM-driven autoresearch loop can automate NER model optimization, revealing narrow viable search spaces on small datasets.

Principles

• Small datasets often have narrow optimization search spaces.
• Autonomous experimentation logs offer valuable failure mode analysis.
• Ensemble methods can boost NER performance.

Method

An LLM-driven coding agent iteratively proposes hypotheses, modifies training configurations, and evaluates against a held-out validation set for Named Entity Recognition.

In practice

• Apply autoresearch to small-data NER tasks.
• Combine DeBERTa-large with MC Dropout.
• Use multi-LLM consensus for robust ensembles.

Topics

• LLM-driven Autoresearch
• Named Entity Recognition
• Opioid Use Data
• Social Media Mining
• DeBERTa-large
• Ensemble Learning

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• How Autoresearch will change Small Language Models adoption — Autoresearch enables autonomous, iterative optimization of ML models by an AI agent editing training code.
• Lazy or Efficient? Towards Accessible Eye-Tracking Event Detection Using LLMs — An LLM-driven pipeline automates eye-tracking event detection, making complex analysis accessible without specialized programming.
• A Reproducible Log-Driven AutoML Framework for Interpretable Pipeline Optimization in Healthcare Risk Prediction — Reproducible, log-driven AutoML reveals healthcare risk prediction pipeline component importance and search space redundancy.
• Analysing Lightweight Large Language Models for Biomedical Named Entity Recognition on Diverse Ouput Formats — Lightweight LLMs offer competitive biomedical NER performance, with specific output formats proving more effective than multi-format tuning.
• ACIL: Auto Chain of Thoughts for In-Context Learning — Auto-CoT improves LLM in-context learning by automatically generating, pruning, and selecting reasoning chains.
• AutoAdapt: Automated domain adaptation for large language models — AutoAdapt automates LLM domain adaptation, transforming a manual, costly process into a reproducible, constraint-aware engineering discipline.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.