SAMPLe: SAM-based Optimizer for Prompt Learning in VLMs

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

SAMPLe (Sharpness-Aware Minimization Prompt Learning) is a novel, model-agnostic optimizer designed to enhance prompt learning in Vision-Language Models (VLMs) like CLIP. It addresses the performance-generalization dilemma by accounting for loss landscape sharpness, balancing exploitation and exploration. SAMPLe dynamically adapts to the current optimization state, reducing overfitting and improving adaptability to unseen data. Integrated into multiple prompt learning frameworks, including CoOp, CoCoOp, MaPLe, TCP, and Co-Prompt, experiments across 11 diverse datasets (e.g., ImageNet, Flowers102, DTD) demonstrate its effectiveness. SAMPLe consistently outperforms existing optimizers, for instance, improving CoOp's harmonic mean from 77.65 to 78.88, showcasing superior generalization to novel classes and robustness in cross-dataset and cross-domain zero-shot settings.

Key takeaway

For AI Scientists and Machine Learning Engineers developing Vision-Language Models, consider integrating SAMPLe to overcome prompt learning's generalization challenges. This optimizer consistently improves performance across diverse datasets and domains by finding flatter, more stable minima. You should evaluate SAMPLe with your existing prompt learning frameworks like CoOp or MaPLe to enhance base-to-new class generalization and cross-domain robustness without sacrificing training accuracy.

Key insights

SAMPLe enhances VLM prompt generalization by optimizing for flatter, lower-loss minima through dynamic sharpness-aware gradient adjustments.

Principles

• Prompt learning faces a performance-generalization trade-off.
• Flatter loss landscapes improve model generalization.
• Dynamic gradient alignment balances exploitation and exploration.

Method

SAMPLe optimizes a dual-objective function, aligning SAM gradient updates with empirical risk minimization (ERM) gradients and ensuring orthogonality to the full-batch gradient, dynamically balancing exploration and exploitation.

In practice

• Integrate SAMPLe into CoOp, CoCoOp, MaPLe frameworks.
• Apply to few-shot recognition tasks for improved generalization.
• Enhance cross-dataset and cross-domain zero-shot transfer.

Topics

• Vision-Language Models
• Prompt Learning
• Sharpness-Aware Minimization
• Model Generalization
• Optimization Algorithms
• CLIP

Best for: Research Scientist, Computer Vision Engineer, AI Scientist, Machine Learning Engineer

Related on AIssential

• Stochastic Gradient Optimization with Model-Assisted Sampling — A model-assisted sampling framework reduces stochastic gradient variance by integrating survey sampling theory with ML optimization.
• Optimizing Deep Learning Models with SAM — Sharpness-Aware Minimization (SAM) improves deep learning model generalizability by addressing overparameterization challenges.
• The Professor: Multi-Teacher Unsupervised Prompt Distillation for Vision-Language Models — Multi-teacher prompt distillation significantly improves VLM student performance, particularly under domain shift, through complementary teacher supervision.
• Minibatch Selection via Partition Matroid Constrained Gradient Matching — PartitionSel optimizes LLM fine-tuning on heterogeneous data by balancing domain coverage and convergence via gradient-matching and matroid constraints.
• Dynamic Cluster Data Sampling for Efficient and Long-Tail-Aware Vision-Language Pre-training — Dynamic cluster-based sampling efficiently trains VLMs, improving long-tail concept performance by balancing data distribution.
• ZEBRA: Zero-Shot Entropy-Regularized Prompt Learning for Base-to-Novel Generalization in Audio-Language Models — Prompt learning in ALMs creates a base-to-novel generalization gap; ZEBRA mitigates this by fusing logits and entropy regularization.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.