Self-supervision drives representational convergence in medical foundation models more than clinical supervision

2026-07-23 · Source: cs.CL updates on arXiv.org · Field: Science & Research — Health & Medical Research, Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

A controlled study dissects representational convergence in medical foundation models, analyzing 18 image and 7 text encoders ranging from 7 million to 27 billion parameters across five imaging modalities, including 650,982 chest radiographs. The research reveals that convergence is modest but real, primarily driven by self-supervised pretraining objectives (40.4% alignment on chest radiography) rather than clinical supervision (21.1%) or image-text objectives (3.3%). This convergence is modality-specific, does not extend to clinical language, and does not scale with model size or performance (Spearman 0.302, p=0.223). Despite its limitations, a linear classifier successfully transfers across encoders and to five external hospitals, retaining about 85% of within-encoder performance. The findings indicate that interoperability in medical imaging models is determined by the pretraining objective, not by scale or clinical supervision.

Key takeaway

For AI Scientists and Machine Learning Engineers developing medical foundation models, you should prioritize self-supervised pretraining objectives to achieve representational convergence and interoperability. This approach enables robust classifier transfer across different encoders and hospital sites, even with modest alignment. Critically, validate shared representations across diverse patient subgroups to ensure equitable and reliable clinical tool deployment.

Key insights

Self-supervision, not clinical labels or scale, drives modest but functional representational convergence in medical foundation models.

Principles

• Self-supervised objectives align medical encoders more effectively.
• Convergence is modality-specific, not cross-modal to language.
• Model scale or performance does not increase convergence.

Method

The study used a controlled training matrix across 18 image and 7 text encoders, varying objectives under fixed data/architecture, and a synthetic generative model to isolate convergence drivers.

In practice

• Design pretraining objectives for interoperability, especially self-supervision.
• Validate shared geometry across patient subgroups for reliability.
• Build encoder-agnostic clinical tools using common coordinate systems.

Topics

• Medical Foundation Models
• Self-Supervised Learning
• Representational Convergence
• Medical Image Encoders
• Cross-Site Transfer
• Patient Subgroup Equity

Code references

• basveeling/pcam
• wisdomikezogwo/quilt1m
• tayebiarasteh/convergence
• mlmed/torchxrayvision

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Self-supervision drives representational convergence in medical foundation models more than clinical supervision — Self-supervised objectives, not scale or clinical supervision, primarily drive representational convergence in medical foundation models.
• LLM4EHR: Aligning Clinical Time Series with Medical Event Sequences via Large Language Models — Aligning clinical time series with medical event sequences via foundation models significantly improves clinical outcome predictions.
• Learning Emergent Modular Representations in Multi-modality Medical Vision Foundation Models — DEX uses modular networks with experts and a director to manage modality-specific and shared representations in multi-modality medical FMs.
• CAP: Towards PPG Universal Representation Learning with Patient-level Supervision — CAP improves PPG representation learning by anchoring signal data to patient-level clinical semantics via cross-modal contrastive pretraining.
• Towards Interpretable Foundation Models for Retinal Fundus Images — Foundation models for medical imaging can achieve high performance and inherent interpretability simultaneously.
• Sasi Kumar Kolla Examines Multimodal Foundation Models for Precision Medicine Research — Multimodal foundation models can integrate diverse biomedical data for precision medicine, overcoming single-modality limitations.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.