Linking spatial biology and clinical histology via Haiku

2026-05-05 · Source: cs.LG updates on arXiv.org · Field: Science & Research — Artificial Intelligence & Machine Learning, Life Sciences & Biology, Health & Medical Research · Depth: Expert, extended

Summary

Haiku is a novel tri-modal contrastive learning model designed to integrate molecular, morphological, and clinical data for biomedical research. Trained on 26.7 million spatial proteomics patches from 3,218 tissue sections across 1,606 patients, it aligns multiplexed immunofluorescence (mIF), hematoxylin and eosin (H&E) histology, and clinical metadata into a shared embedding space. Haiku enables three-way cross-modal retrieval, significantly outperforming unimodal baselines with Recall@50 up to 0.611. It also improves downstream classification and clinical prediction tasks, achieving a C-index of 0.737 for survival prediction and a mean Pearson correlation of 0.718 for zero-shot biomarker inference. Furthermore, Haiku supports a counterfactual prediction framework, revealing niche-specific molecular shifts associated with breast cancer stage progression and lung cancer survival outcomes by modifying clinical metadata while fixing tissue morphology.

Key takeaway

For AI scientists and machine learning engineers developing computational pathology solutions, Haiku offers a robust framework for integrating diverse biomedical data. You should consider adopting its tri-modal contrastive learning approach to enhance cross-modal retrieval, improve clinical prediction accuracy, and enable exploratory counterfactual analyses. This model's ability to ground biomarker inference in real mIF patches provides a verifiable alternative to purely generative methods, making it valuable for hypothesis generation and translational research.

Key insights

Haiku is a tri-modal contrastive learning model that unifies spatial proteomics, H&E histology, and clinical text into a shared embedding space.

Principles

• Tri-modal alignment improves cross-modal retrieval and downstream prediction.
• Counterfactual perturbations reveal niche-specific molecular shifts.
• Evidence-based retrieval grounds biomarker predictions in real data.

Method

Haiku uses modality-specific encoders (MUSK for H&E, VirTues for mIF, BiomedBERT for text) with projection heads, trained via a tri-modal contrastive loss to align embeddings in a shared latent space.

In practice

• Use Haiku for cross-modal retrieval of mIF, H&E, or text data.
• Apply Haiku for zero-shot biomarker inference using H&E and clinical text.
• Explore counterfactual scenarios by perturbing clinical metadata.

Topics

• Haiku Model
• Tri-modal Contrastive Learning
• Spatial Proteomics
• Clinical Histology
• Cross-modal Retrieval

Code references

• zhihuanglab/Haiku

Best for: AI Scientist, Research Scientist, Machine Learning Engineer

Related on AIssential

• A generalist biomedical vision-language model via multi-CLIP knowledge distillation — MMKD-CLIP effectively builds a generalist biomedical foundation model via multi-CLIP knowledge distillation for diverse medical tasks.
• MLLM-Microscope: Unlocking Hidden Structure Within Multimodal Large Language Models — MLLM-Microscope reveals MLLM internal representation dynamics, showing modality fusion impacts linearity, dimension, and anisotropy across transformer layers.
• Virtual 3D H&E Staining from Phase-contrast Back-illumination Interference Tomography — A novel dataset and GAN framework enable accurate 3D virtual H&E staining from label-free Back-illumination Interference Tomography.
• A Triple-Modal Contrastive Learning Framework with Sequence, Graph, and 3D Features for Drug-Target Interaction Prediction — TriMod-DTI uses triple-modal contrastive learning with 1D, 2D, and 3D features for superior drug-target interaction prediction.
• CellNiche represents cellular microenvironments in atlas-scale spatial omics data with contrastive learning — CellNiche uses contrastive learning on spatial omics data to map cellular microenvironments across diverse biological contexts.
• Dual-Modal Lung Cancer AI: Interpretable Radiology and Microscopy with Clinical Risk Integration — Integrating CT radiology and H&E microscopy with clinical data significantly enhances lung cancer diagnosis and interpretability.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.LG updates on arXiv.org.