Rethinking Noise-Robust Training for Frozen Vision Foundation Models: A Cross-Dataset Benchmark with a Case Study of Small-Loss Failure

2026-05-21 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision & Pattern Recognition · Depth: Expert, quick

Summary

A new benchmark evaluates eight noisy-label learning methods for frozen Vision Foundation Models (VFMs) used in medical imaging, addressing the gap in understanding these methods for frozen-feature regimes. The study, involving 150 conditions and 6,000 training runs across five medical datasets, three backbones, two noise types, and five noise rates, found no universally superior method. While ELR won the most conditions (49/150), CUFIT achieved the best mean rank (2.51). The practical cost of selecting the wrong method escalates significantly with noise severity, from 4.5 percentage points on clean data to 18.8 percentage points at asymmetric 40% noise. The research also reveals that under frozen DINOv2 features, clean and noisy loss distributions overlap by 53-61%, challenging the traditional "small-loss assumption." This suggests noisy-label learning for frozen VFMs requires regime-aware method selection rather than seeking a single dominant algorithm.

Key takeaway

For Machine Learning Engineers deploying frozen Vision Foundation Models in medical imaging, you should critically assess noisy-label learning methods based on specific noise regimes rather than assuming a single best algorithm. Your method choice significantly impacts performance, with incorrect selections costing up to 18.8 percentage points in balanced accuracy under severe asymmetric noise. Prioritize regime-aware selection and consider the provided evidence-based guidance to avoid substantial performance degradation, especially for minority classes.

Key insights

No universal noisy-label learning method exists for frozen Vision Foundation Models; selection must be regime-aware.

Principles

• The "small-loss assumption" is unreliable for frozen VFM noisy-label learning.
• Method choice cost increases sharply with noise severity.
• Prediction agreement is more stable than loss ranking under asymmetric noise.

Method

The article describes a controlled benchmark of eight noisy-label methods across 150 conditions, evaluating balanced accuracy to identify performance patterns.

In practice

• Prioritize regime-aware method selection over seeking a single best algorithm.
• Re-evaluate small-loss assumption for frozen VFM training.
• Consider prediction agreement stability for noise detection.

Topics

• Vision Foundation Models
• Noisy Label Learning
• Medical Imaging
• DINOv2
• Balanced Accuracy
• Machine Learning Benchmarking

Best for: Computer Vision Engineer, AI Scientist, Research Scientist, Machine Learning Engineer

Related on AIssential

• Federated Medical Image Segmentation under Real-World Label Noise: A Benchmark Suite for Noisy Label Learning Method Selection — The new benchmark suite enables realistic evaluation and selection of Federated Noisy Label Learning methods for medical image segmentation.
• Evaluating the role of pretraining dataset size and diversity on single-cell foundation model performance — Single-cell foundation models do not show clear data scaling laws, unlike large language models.
• Multi-Label Test-Time Adaptation with Bayesian Conditional Priors — BCP enhances multi-label VLM recognition under distribution shift by injecting label dependency via anchor-conditioned Bayesian refinement.
• Conformal Margin Risk Minimization: An Envelope Framework for Robust Learning under Label Noise — CMRM improves noisy label learning by using a quantile-calibrated regularization term without privileged knowledge.
• Realistic noise synthesis reduces bias and improves tissue microstructure estimation with supervised machine learning — Realistic noise synthesis in simulated data is crucial for unbiased dMRI microstructure estimation, especially at low SNR.
• Active Learning with Foundation Model Priors: Efficient Learning under Class Imbalance — The framework uses foundation model priors for imbalance-aware active learning, achieving over 50% annotation savings in noisy, imbalanced datasets.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.