PRiSM: Prototype Regularization for Few-Shot VLMs

2026-07-23 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, extended

Summary

PRiSM is a novel prototype regularization framework designed to enhance few-shot adaptation in Vision-language Models (VLMs) under realistic, class-imbalanced conditions. The research introduces a new benchmark using Dirichlet sampling to vary class balance and effective class numbers, revealing that existing training-free methods suffer significant performance degradation, often worsening with more labeled samples due to biased class prototypes. PRiSM addresses this by acting as a plug-and-play module, optimizing a multi-term loss that maximizes inter-class pairwise distances, aligns support features, and maintains fidelity to baseline prototypes. It employs an efficient block Majorize-Minimize optimizer, leveraging a Lipschitz constant derived from the Gershgorin circle theorem. Experiments across 10 diverse datasets demonstrate PRiSM's consistent and substantial improvements for baselines like Tip-Adapter, APE, GDA, and ProKeR, with Tip-Adapter seeing a +35.05% mean accuracy gain in 16-shot severe imbalance.

Key takeaway

For Machine Learning Engineers developing few-shot VLM applications, recognize that standard class-balanced evaluation protocols are unrealistic. Real-world data imbalance severely degrades VLM performance, often worsening with increased labeled samples due to prototype bias. You should integrate prototype regularization frameworks like PRiSM as a plug-and-play module to rectify these biases. This approach ensures robust performance in challenging, high-class, and imbalanced scenarios, significantly boosting weaker baselines without requiring extensive hyperparameter tuning.

Key insights

Prototype bias from class imbalance severely degrades few-shot VLM performance; PRiSM rectifies this with novel regularization.

Principles

• Realistic few-shot benchmarks must account for class imbalance.
• Prototype bias, not adaptation mechanism, often limits VLM performance.
• Increasing samples under imbalance can amplify bias, degrading performance.

Method

PRiSM optimizes a multi-term loss combining support-feature alignment, baseline prototype fidelity, and inter-class distance maximization, using a block Majorize-Minimize algorithm with a Lipschitz-driven step size.

In practice

• Deploy PRiSM as a plug-and-play module on existing VLM baselines.
• Use LoRA for efficient vision encoder parameter adaptation.
• Apply Dirichlet sampling for realistic few-shot dataset construction.

Topics

• Vision-language Models
• Few-shot Learning
• Class Imbalance
• Prototype Regularization
• Majorize-Minimize Optimization
• CLIP
• Dirichlet Sampling

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Supervised Classification Heads as Semantic Prototypes: Unlocking Vision-Language Alignment via Weight Recycling — Repurposing pretrained vision model classification heads as semantic prototypes enables efficient VLM alignment and data augmentation.
• A Quiet Failure in Calibrated Virtual Screening: Marginal Conformal Prediction Under-Covers the Minority Class, and a Class-Conditional Fix Recovers It — Marginal conformal prediction under-covers minority classes on imbalanced datasets, an invisible failure remedied by class-conditional calibration.
• When Vision-Language Models Judge Without Seeing: Exposing Informativeness Bias — VLM-as-a-Judge systems often prioritize informative answers over image-grounded correctness, leading to "informativeness bias."
• Alleviating Regional Shortcuts for Few-Shot Class-Incremental Learning — Models in FSCIL develop "regional shortcuts," over-focusing on base-class features and hindering novel-class adaptation.
• SAMPLe: SAM-based Optimizer for Prompt Learning in VLMs — SAMPLe enhances VLM prompt generalization by optimizing for flatter, lower-loss minima through dynamic sharpness-aware gradient adjustments.
• Embracing Intra-Class Heterogeneity for Semi-Supervised Medical Image Segmentation: From Diversity to Precision — Explicitly modeling intra-class heterogeneity with multiple intensity-aligned prototypes improves semi-supervised medical image segmentation precision, especially with limited data.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.