VaxScope: Document-Level Structured Evidence Extraction from Immunization Systematic Reviews

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Health & Medical Research · Depth: Expert, medium

Summary

VaxScope is a new benchmark dataset designed for document-level structured evidence extraction from immunization-related systematic reviews, introduced at BioNLP 2026. This dataset addresses the challenge of extracting clinical evidence from systematic reviews, where information is often distributed across multiple studies and sections, making traditional span-level NLP methods insufficient. VaxScope was created using an expert-guided semi-automatic annotation pipeline, combining automatic candidate generation with domain expert validation to ensure high quality. The task is formalized as document-level structured extraction, requiring aggregation of evidence beyond isolated text spans. Baseline experiments demonstrated that PubMedBERT achieved the best overall performance with an average F1 score of 0.850. Crucially, integrating evidence-grounded contextual input significantly improved performance, particularly for tasks demanding distributed contextual reasoning, compared to abstract-only settings.

Key takeaway

For NLP Engineers developing evidence extraction systems for biomedical systematic reviews, recognize that traditional span-level methods are inadequate. You should prioritize document-level structured extraction, aggregating evidence across multiple sections and studies. Utilize models like PubMedBERT, especially by incorporating evidence-grounded contextual input, to significantly improve performance on fields requiring distributed contextual reasoning. This approach will enhance the accuracy and utility of your systems for complex medical literature.

Key insights

Document-level structured evidence extraction from systematic reviews requires aggregating distributed information beyond isolated text spans.

Principles

• Systematic review evidence necessitates document-level aggregation.
• Distributed contextual reasoning enhances evidence extraction.
• Expert-guided semi-automatic annotation improves data quality.

Method

An expert-guided semi-automatic annotation pipeline combines automatic candidate generation with domain expert validation to create high-quality document-level structured extraction datasets.

In practice

• Apply PubMedBERT for document-level evidence extraction in biomedical NLP.
• Integrate evidence-grounded contextual input to boost extraction performance.

Topics

• VaxScope
• Document-Level Extraction
• Systematic Reviews
• Immunization
• Biomedical NLP
• PubMedBERT

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• AAbAAC: An Annotated Corpus for Autoimmunity Information Extraction — Small, specialized annotated corpora significantly boost information extraction performance in niche biomedical domains.
• BCL: Bayesian In-Context Learning Framework for Information Extraction — BCL uses Bayesian particle filtering to systematically optimize in-context learning for information extraction, improving performance and generalizability.
• ExStrucTiny: A Benchmark for Schema-Variable Structured Information Extraction from Document Images — ExStrucTiny is a new benchmark for structured information extraction from diverse, schema-variable document images.
• EHRNote-ChatQA: A Benchmark for Evidence-Grounded Multi-Turn Clinical Question Answering over Longitudinal Discharge Summaries — EHRNote-ChatQA benchmarks LLMs on evidence-grounded, multi-turn clinical QA over longitudinal discharge summaries, revealing current limitations.
• Configurable Clinical Information Extraction with Agentic RAG: What Works, What Breaks, and Why — Agentic RAG effectively extracts clinical information from complex, metadata-poor patient contexts, achieving 96.5% clinician acceptance.
• Measuring the sensitivity of LLM-based structured extraction to prompt, model, and schema choices in clinical discharge summaries — LLM-based clinical data extraction sensitivity varies significantly with model and schema, especially regarding "absence-versus-silence" distinctions.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.