Beyond Knowledge Graphs: PubMedBERT Embeddings as a Competitive Standalone Modality for Drug Re-purposing

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Science & Research — Life Sciences & Biology, Health & Medical Research, Mathematics & Computational Sciences · Depth: Expert, quick

Summary

Kondadadi and Ortega's research demonstrates that PubMedBERT text embeddings offer a competitive standalone modality for drug repurposing, challenging the heavy reliance on knowledge graph (KG) embeddings. On the Hetionet drug-disease benchmark, PubMedBERT achieved an AUROC of 0.910, surpassing four re-trained KG baselines (TransE, ComplEx, DistMult, RotatE), where RotatE was the best at 0.854. A Random Forest model using these embeddings scored 0.880. The authors note this comparison is "text-with-literature-supervision vs. graph-only." Furthermore, attempts to fuse text with molecular (ECFP4) and gene expression (LINCS L1000) features via cross-attention consistently degraded performance, indicating issues with noise absorption. The study also revealed that text embeddings encode association strength but not direction, leading to proconvulsants like amoxapine and flumazenil being highly ranked.

Key takeaway

For AI Scientists developing drug repurposing models, you should prioritize evaluating standalone PubMedBERT embeddings as a strong baseline before investing heavily in complex knowledge graph construction. Your efforts in multimodal fusion, particularly with cross-attention, might degrade performance rather than improve it; carefully validate fusion strategies. Additionally, ensure your models explicitly account for association directionality to avoid misidentifying adverse compounds as therapeutic candidates.

Key insights

PubMedBERT embeddings alone outperform knowledge graph baselines for drug repurposing, but multimodal fusion can degrade performance.

Principles

• Text embeddings can rival KG embeddings.
• Multimodal fusion requires careful noise handling.
• Association strength differs from directionality.

Method

The study used PubMedBERT text embeddings with downstream classifiers on 10-fold splits, comparing against re-trained KG baselines (TransE, ComplEx, DistMult, RotatE) and a Random Forest. Cross-attention fused modalities.

In practice

• Evaluate standalone text embeddings before complex KGs.
• Scrutinize multimodal fusion for performance degradation.
• Validate drug repurposing models for directionality.

Topics

• Drug Repurposing
• PubMedBERT
• Knowledge Graph Embeddings
• Multimodal Fusion
• Text Embeddings
• Hetionet Benchmark

Best for: NLP Engineer, AI Scientist, Research Scientist

Related on AIssential

• GraphDiffMed: Knowledge-Constrained Differential Attention with Pharmacological Graph Priors for Medication Recommendation — Combining dual-scale differential attention with pharmacological graph priors significantly enhances medication recommendation safety and quality.
• Correlation Is Not Enough: Embedding Human Metadata for Individual Causal Discovery — Pretrained biomedical language models struggle with cross-domain causal inference due to false embedding proximity, requiring specialized contrastive training.
• Representation learning to advance multi-institutional studies with electronic health record data from US and France — A graph-based framework harmonizes heterogeneous EHR data across institutions using representation learning and privacy-preserving methods.
• Cross-Domain Feature Expansion for Tabular Medical Data via Knowledge Graphs Injection — MedKGTab expands tabular medical features by combining data-driven statistics with biomedical knowledge graphs for high-fidelity generation.
• SPOKE: A massive biomedical knowledge graph for precision health and drug discovery — SPOKE integrates vast biomedical data into a knowledge graph for drug discovery and precision medicine.
• Multi-to-uni modal knowledge transfer pre-training for molecular representation learning — M2UMol enables robust molecular representation learning from incomplete multimodal data by transferring knowledge to a 2D encoder.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.