A Comparative Analysis of In-Context Learning and Fine-Tuning for Biomedical Information Retrieval and Sentence Extraction Using Research Domain Criteria

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Science & Research — Health & Medical Research, Mathematics & Computational Sciences, Research Methodology & Innovation · Depth: Advanced, quick

Summary

A recent study compared domain-adapted fine-tuning and in-context prompting for two Research Domain Criteria (RDoC)-related subtasks from the BioNLP-OST 2019 RDoC shared task. The RDoC framework, from the National Institute of Mental Health, integrates information across genetics, circuits, and behavior for mental disorder studies. Task 1 involved retrieving and ranking unlabeled PubMed abstracts for eight RDoC constructs, comparing a TF-IDF baseline against ModernBERT and Llama (zero-shot and five-shot) using Mean Average Precision (MAP). Task 2 focused on identifying the single most relevant sentence from an abstract for a given construct, evaluated by per-construct accuracy. For fine-tuning, BioBERT, PubMedBERT, ModernBERT, and RoBERTa were used with a cross-encoder input format and per-construct grid search. These were benchmarked against in-context learning with several open-source language models. Both strategies achieved competitive results against the top BioNLP-OST 2019 team, suggesting that five-shot prompted LLMs and domain-adapted fine-tuned transformers are viable for semi-automating RDoC curation.

Key takeaway

For Research Scientists tasked with semi-automating RDoC curation, you should consider both domain-adapted fine-tuning and five-shot in-context learning. Your choice depends on resource availability and specific task requirements. Fine-tuned transformers like BioBERT or PubMedBERT offer robust performance for abstract retrieval and sentence extraction. Alternatively, five-shot prompted LLMs, such as Llama, provide competitive results, potentially reducing extensive training data needs for similar biomedical information retrieval challenges.

Key insights

Both fine-tuning and five-shot in-context learning are effective for RDoC biomedical text tasks.

Principles

• Domain adaptation enhances model performance.
• In-context learning scales with few-shot examples.
• Cross-encoder format improves relevance ranking.

Method

Compares domain-adapted fine-tuning (BioBERT, PubMedBERT, ModernBERT, RoBERTa with cross-encoder) against in-context prompting (Llama, other LLMs) for abstract retrieval and sentence extraction.

In practice

• Use five-shot prompting for LLM-based RDoC tasks.
• Apply cross-encoder fine-tuning for biomedical relevance.
• Consider ModernBERT or Llama for abstract ranking.

Topics

• Research Domain Criteria
• Biomedical Information Retrieval
• In-Context Learning
• Fine-Tuning
• Large Language Models
• Sentence Extraction

Best for: NLP Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Improving Answer Extraction in Context-based Question Answering Systems Using LLMs — Fine-tuning pre-trained LLMs on benchmark QA datasets significantly improves answer extraction accuracy and contextual understanding in QA systems.
• BioHiCL: Hierarchical Multi-Label Contrastive Learning for Biomedical Retrieval with MeSH Labels — Hierarchical MeSH labels provide fine-grained supervision for biomedical dense retrieval, improving semantic representation.
• Domain Fine-Tuning FinBERT on Finnish Histopathological Reports: Train-Time Signals and Downstream Correlations — Domain fine-tuning FinBERT on medical text can be predicted by observing embedding geometry changes.
• Benchmarking Retrieval Strategies for Biomedical Retrieval-Augmented Generation: A Controlled Empirical Study — Cross-Encoder Reranking significantly improves biomedical RAG performance by enhancing contextual precision.
• EQ-5D Classification Using Biomedical Entity-Enriched Pre-trained Language Models and Multiple Instance Learning — Entity enrichment significantly enhances PLM performance for domain-specific text classification in biomedical literature.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers — Finetuning multimodal embedding models on domain-specific data significantly boosts performance over general-purpose models.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.