Label Shift Aware Adaptation for Online Zero-shot Learning with Contrastive Language-Image Pre-Training (CLIP)

2026-06-13 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, quick

Summary

Label Shift Aware (LSA) is a novel approach designed for online zero-shot learning with Contrastive Language-Image Pre-Training (CLIP) models. This method addresses the performance degradation that occurs when the label distribution of test data differs from CLIP's initial training domain, a common issue in scenarios where CLIP's feature extraction and model parameters remain fixed during sequential inference. LSA frames the online zero-shot classification task as a domain adaptation problem. It specifically adapts predictions generated by CLIP, which was trained on an unknown source distribution, to a target distribution using only unlabeled test data. By applying label shift correction, LSA effectively mitigates the mismatch between source and target domains. Extensive experiments across multiple datasets demonstrate that LSA consistently outperforms existing state-of-the-art online zero-shot learning methods based on CLIP.

Key takeaway

For Computer Vision Engineers deploying CLIP in online zero-shot learning scenarios, recognizing and addressing label distribution shifts is crucial. Your sequential inference performance can degrade significantly when test data differs from CLIP's training domain. Implementing label shift correction, as demonstrated by LSA, offers a robust strategy to adapt CLIP predictions using only unlabeled test data, thereby maintaining accuracy and improving reliability in challenging, data-scarce environments.

Key insights

Label Shift Aware (LSA) enhances online zero-shot CLIP performance by explicitly correcting for label distribution shifts between source and target domains.

Principles

• Label distribution mismatch degrades online zero-shot CLIP.
• Frame online zero-shot as a domain adaptation problem.
• Apply label shift correction for domain mismatch.

Method

LSA adapts CLIP predictions from an unknown source distribution to a target using unlabeled test data. It applies label shift correction to mitigate the mismatch between source and target domains.

Topics

• CLIP
• Zero-shot Learning
• Online Learning
• Domain Adaptation
• Label Shift Correction
• Computer Vision

Best for: Research Scientist, AI Scientist, Computer Vision Engineer

Related on AIssential

• Improving CLIP Adaptation by Breaking Tail Alignment for Source-Free Cross-Domain Few-Shot Learning — Breaking uniform alignment for low-similarity "tail tokens" improves CLIP adaptation in cross-domain few-shot learning.
• Multi-Label Test-Time Adaptation with Bayesian Conditional Priors — BCP enhances multi-label VLM recognition under distribution shift by injecting label dependency via anchor-conditioned Bayesian refinement.
• Characterizing and Correcting Effective Target Shift in Online Learning — Online learning's sub-optimality stems from effective target shifts, which can be corrected to match offline performance.
• ZeroDiff++: Substantial Unseen Visual-semantic Correlation in Zero-shot Learning — ZeroDiff++ enhances zero-shot learning by mitigating spurious visual-semantic correlations through diffusion-based generation and adaptation.
• Calibrating Uncertainty for Zero-Shot Adversarial CLIP — Adversarial attacks on CLIP suppress uncertainty, leading to over-confident misclassifications; calibrating this is crucial for reliability.
• Conformal Bayes under Label Shift: Post-Hoc Calibration vs. In-Training Adaptation — Conformal Bayes under label shift can be adapted via post-hoc calibration or in-training methods to maintain valid coverage.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.