Training Biomedical Retrievers From Large-Scale Citation Contexts

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Science & Research — Health & Medical Research, Artificial Intelligence & Machine Learning, Research Methodology & Innovation · Depth: Expert, short

Summary

The CiteRec model, a lightweight BERT-based retriever-reranker, has been developed to train effective biomedical retrievers using freely available citation sentences. Researchers constructed a large-scale training dataset of approximately 62 million citation sentence-abstract pairs extracted from PubMed Central. This model was evaluated across three benchmark settings: the biomedical subset of BEIR for information retrieval, SciRepEval for generalizable scientific document embeddings, and CitancePlus, a new dataset of about 90 thousand citation sentence-abstract pairs for PubMed-scale citation recommendation. CiteRec demonstrated competitive performance against the MedCPT model on the biomedical BEIR subset and surpassed MedCPT on SciRepEval. Furthermore, CiteRec achieved strong performance for citation recommendation across the entire PubMed corpus, outperforming both MedCPT and the substantially larger Qwen3-Embedding-8B retriever.

Key takeaway

For machine learning engineers developing biomedical information retrieval systems, this research indicates that relying solely on proprietary data is unnecessary. You can achieve competitive or superior performance by using freely available citation contexts from sources like PubMed Central. Consider training lightweight BERT-based models such as CiteRec on large-scale public datasets to reduce costs and improve accessibility. This approach offers a viable alternative to larger, resource-intensive models.

Key insights

Freely available citation contexts can train biomedical retrievers competitive with or superior to those trained on proprietary data.

Principles

• Open-source data can rival proprietary datasets.
• Lightweight models can outperform larger ones.
• Citation contexts are rich for retriever training.

Method

Construct a ~62 million citation sentence-abstract pair dataset from PubMed Central. Train a BERT-based retriever-reranker model, CiteRec, on this dataset. Evaluate on BEIR, SciRepEval, and CitancePlus benchmarks.

In practice

• Use PubMed Central for large-scale training data.
• Consider BERT-based models for biomedical retrieval.
• Evaluate on diverse benchmarks like CitancePlus.

Topics

• Biomedical Information Retrieval
• Citation Recommendation
• BERT-based Models
• PubMed Central
• Document Embeddings
• SciRepEval

Best for: NLP Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Benchmarking Retrieval Strategies for Biomedical Retrieval-Augmented Generation: A Controlled Empirical Study — Cross-Encoder Reranking significantly improves biomedical RAG performance by enhancing contextual precision.
• KliniskVestBERT: BERT Model Specialised to Norwegian Clinical Texts — Domain-specific pre-training significantly enhances BERT models for Norwegian clinical NLP tasks.
• BioHiCL: Hierarchical Multi-Label Contrastive Learning for Biomedical Retrieval with MeSH Labels — Hierarchical MeSH labels provide fine-grained supervision for biomedical dense retrieval, improving semantic representation.
• Nested Named Entity Recognition in Plasma Physics Research Articles — A lightweight BERT-CRF model with entity-specific specialization and Bayesian Optimization effectively extracts nested entities from plasma physics texts.
• Few-Shot Biomedical Relation Extraction with Large Language Models: A Viable Alternative to Supervised Learning? — LLMs offer a viable few-shot alternative for biomedical relation extraction, outperforming prior few-shot methods, especially for rare relations.
• MiqraBERT: Regression-Based Sentence-BERT Finetuning for Biblical Hebrew Parallel Detection — Finetuning a BERT model with regression for semantic similarity effectively detects textual reuse in ancient languages beyond lexical overlap.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.