Citation-Aware Continual Pre-Training for Biomedical Language Models

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Science & Research — Health & Medical Research, Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A novel citation-aware continual pre-training method for decoder-only language models incorporates citation graph information from PubMed into next-token prediction. This approach, proposed by Masaki Asada et al., places citation-linked abstract pairs within a shared context, a technique typically ignored in standard language model pre-training despite the rich structured knowledge in biomedical literature. Evaluated on multiple biomedical QA benchmarks using two model families, the method demonstrates higher average accuracy. These results surpass both the original base models and models subjected to citation-unaware pre-training across various biomedical tasks, highlighting the value of integrating relational data.

Key takeaway

For AI Scientists developing biomedical language models, incorporating citation graph information via continual pre-training significantly boosts accuracy on QA benchmarks. You should consider this method to enhance model performance and capture structured knowledge from scientific literature, especially when working with decoder-only architectures. This approach offers a clear path to improving the utility of LMs in specialized domains.

Key insights

Incorporating citation graph data into language model pre-training significantly improves biomedical QA accuracy.

Principles

• Citation links encode valuable relationships between scientific studies.
• Placing citation-linked abstracts in shared context enhances next-token prediction.

Method

Citation-aware continual pre-training for decoder-only LMs integrates PubMed citation graph data by placing linked abstract pairs in a shared context for next-token prediction.

In practice

• Apply citation-aware pre-training to enhance biomedical QA models.
• Use PubMed citation graphs for domain-specific LM fine-tuning.

Topics

• Biomedical Language Models
• Continual Pre-training
• Citation Graphs
• PubMed
• Question Answering
• Next-Token Prediction

Best for: NLP Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Can Continual Pre-training Bridge the Performance Gap between General-purpose and Specialized Language Models in the Medical Domain? — Continual pre-training and merging specialized 7B LLMs can rival larger general-purpose models in specific domains.
• Knowledge Graphs and Reasoning LLMs for Finding Simple Yet Effective Transcriptomic Perturbation Predictors — Simple K-nearest neighbour models leveraging knowledge graphs, enhanced by RL-trained LLMs, offer competitive transcriptomic perturbation prediction.
• Less is More: Quality-Aware Training Data Selection for Scientific Summarization — Selecting high-quality reference summaries significantly enhances scientific summarization model training efficiency and factuality.
• Stanford AI Lab Papers and Talks at ACL 2022 — ACL 2022 showcased diverse NLP research from Stanford, spanning model pretraining, analysis, and evaluation.
• EQ-5D Classification Using Biomedical Entity-Enriched Pre-trained Language Models and Multiple Instance Learning — Entity enrichment significantly enhances PLM performance for domain-specific text classification in biomedical literature.
• Teaching AI to Remember: What I Learned Building a Continual Learning Framework for Drug Discovery — Continual learning models can avoid catastrophic forgetting by integrating new knowledge with awareness of existing representations.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.