Self-supervision drives representational convergence in medical foundation models more than clinical supervision

2026-07-22 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Medical Imaging AI · Depth: Expert, quick

Summary

A study dissects representational convergence in medical foundation models, analyzing 18 image and 7 text encoders ranging from 7M to 27B parameters across five imaging modalities, including 650,982 chest radiographs. It reveals that convergence is modest but consistently above random, primarily driven by the self-supervised objective (40.4% alignment on chest radiography) rather than clinical supervision (21.1%) or image-text objectives (3.3%). This convergence does not increase with model size or capability and is limited to within-modality contexts, failing to align with clinical language or radiologist judgment. Despite these limitations, a linear classifier successfully transfers across different encoders and to five held-out hospitals, retaining approximately 85% of its within-encoder performance.

Key takeaway

For AI Scientists designing medical foundation models, prioritize self-supervised pretraining objectives to enhance representational convergence and interoperability. While linear classifiers show promising transferability (retaining ~85% performance), you must rigorously validate the shared geometry across diverse patient subgroups and against actual clinical judgment to ensure robust and equitable real-world application. Your design choices for pretraining objectives directly impact model generalizability and clinical utility.

Key insights

Self-supervised objectives, not scale or clinical supervision, primarily drive representational convergence in medical foundation models.

Principles

• Representational convergence in medical imaging is modest.
• Self-supervision is the primary driver of convergence.
• Convergence does not scale with model size or capability.

Method

A controlled dissection across 18 image and 7 text encoders, varying only the objective under fixed data, architecture, and scale, and reproducing the effect in a synthetic model.

In practice

• Linear classifiers transfer across diverse encoders.
• Performance retains ~85% on held-out hospital data.

Topics

• Medical Foundation Models
• Self-supervised Learning
• Representational Convergence
• Medical Imaging
• Clinical Supervision
• Model Interoperability

Best for: Computer Vision Engineer, AI Scientist, Research Scientist, Machine Learning Engineer

Related on AIssential

• Self-supervision drives representational convergence in medical foundation models more than clinical supervision — Self-supervision, not clinical labels or scale, drives modest but functional representational convergence in medical foundation models.
• Towards Interpretable Foundation Models for Retinal Fundus Images — Foundation models for medical imaging can achieve high performance and inherent interpretability simultaneously.
• Google’s SensorFM turns messy wearable sensor data into a general-purpose health intelligence layer — SensorFM uses massive unlabeled wearable data to create a general foundation model for diverse health and behavioral tasks.
• DaX: Learning General Pathology Representations Across Scales — DaX is a pathology vision foundation model using DINOv3-style self-supervised learning for robust, transferable representations across diverse histopathology tasks.
• Learning Emergent Modular Representations in Multi-modality Medical Vision Foundation Models — DEX uses modular networks with experts and a director to manage modality-specific and shared representations in multi-modality medical FMs.
• Learning neuroimaging models from health system-scale data — Health system-scale data enables AI foundation models to achieve superior, fair, and explainable neuroimaging diagnostics.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.