SegTME-UNI2: A Foundation Model-Based Framework for Generalisable Multiclass Cell Segmentation and LLM-Driven Tumour Microenvironment Characterisation in Histopathology

2026-06-16 · Source: Computer Vision and Pattern Recognition · Field: Science & Research — Health & Medical Research, Artificial Intelligence & Machine Learning, Life Sciences & Biology · Depth: Expert, quick

Summary

SegTME-UNI2 is a unified framework for generalizable multiclass cell segmentation and LLM-driven tumour microenvironment (TME) characterization in H&E-stained histology images. Its core, UNI2-UPERHOVER, is a dual-head segmentation model combining the UNI2-H pathology foundation model (ViT-Giant, pretrained on >100M tiles from 100K slides) with UperNet decoders for six-class semantic segmentation and nuclear instance separation. To address annotation scarcity, UNI2-UPERHOVER uses a three-stage progressive pseudo-label curriculum. This starts with human-annotated PanNuke (7,901 images, 189,744 nuclei, 0.25 um/pixel) and progresses to entropy-filtered pseudo-labels on 271,711 TCGA-UT scale-0 patches (0.5 um/pixel), then all 1,608,060 TCGA-UT patches across six resolution scales (0.5-1.0 um/pixel). The framework extracts over 20 TME features, encoded as JSON, which a fine-tuned NVIDIA BioNeMo GPT model uses to generate clinical narratives. Validation on held-out data confirms feasibility, and the dataset and checkpoint are publicly released.

Key takeaway

For computational pathologists or AI scientists developing TME analysis tools, SegTME-UNI2 offers a validated framework to overcome annotation limitations and generate interpretable clinical reports. You should consider integrating its progressive pseudo-labeling curriculum to scale your segmentation models on large, unannotated datasets. Utilizing the publicly released UNI2-UPERHOVER checkpoint and pseudo-labelled TCGA-UT dataset can accelerate your research in spatial biology and TME profiling.

Key insights

SegTME-UNI2 integrates foundation models and pseudo-labeling for robust, generalizable multiclass cell segmentation and TME characterization.

Principles

• Foundation models improve pathology segmentation.
• Progressive pseudo-labeling scales annotation.
• Dual-head decoders enable multi-task segmentation.

Method

UNI2-UPERHOVER uses a three-stage progressive pseudo-label curriculum on PanNuke and TCGA-UT, followed by 20+ TME feature extraction and JSON encoding for NVIDIA BioNeMo GPT narrative generation.

In practice

• Utilize UNI2-UPERHOVER for cell segmentation.
• Employ pseudo-labeling for large datasets.
• Integrate LLMs for TME report generation.

Topics

• Tumour Microenvironment Characterization
• Multiclass Cell Segmentation
• Pathology Foundation Models
• Progressive Pseudo-labeling
• Histology Image Analysis
• NVIDIA BioNeMo GPT

Best for: Computer Vision Engineer, AI Scientist, Research Scientist, Machine Learning Engineer

Related on AIssential

• SegGuidedNet: Sub-Region-Aware Attention Supervision for Interpretable Brain Tumor Segmentation — SegGuidedNet achieves accurate brain tumor sub-region segmentation with built-in interpretability through sub-region-aware attention supervision.
• Mutual Distillation of Dual-Foundation Models for Semi-Supervised PET/CT Segmentation — Mutual distillation of dual-foundation models improves semi-supervised PET/CT segmentation with minimal labeled data.
• Atlas H&E-TME: Scalable AI-Based Tissue Profiling at Expert Pathologist-Level Accuracy — Atlas H&E-TME provides scalable, expert-level AI-based tissue profiling from H&E slides, validated by a robust dual framework.
• Vision-Language Models as Zero-Annotation Oracles in Histopathology — General-purpose Vision-Language Models (VLMs) offer robust, zero-annotation foreground segmentation in histopathology, generalizing across diverse stains.
• Non-invasive profiling of the tumour microenvironment with spatial ecotypes — A new machine learning framework enables non-invasive profiling of tumor microenvironment spatial ecotypes from liquid biopsies, predicting immunotherapy response.
• A universal foundation model for grounded biomedical image interpretation — UniBiomed unifies diagnostic findings and target segmentation in biomedical images, enhancing AI interpretability for clinical use.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.