Ramen: Robust Test-Time Adaptation of Vision-Language Models with Active Sample Selection

2026-04-23 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, quick

Summary

Ramen is a novel framework designed for robust test-time adaptation of vision-language models (VLMs) like CLIP, specifically addressing performance degradation under mixed-domain distribution shifts. Unlike prior methods that assume consistent test domains, Ramen actively selects relevant samples from previously seen data for each incoming test sample. This selection is based on two criteria: domain consistency, to focus adaptation on similar domains, and prediction balance, to counteract adaptation bias from skewed predictions. To enhance efficiency, Ramen utilizes an embedding-gradient cache, storing embeddings and sample-level gradients of past images. This cache enables retrieval of relevant samples and aggregation of gradients for model updates without requiring additional forward or backward passes. Theoretical analysis supports its effectiveness, and experiments on various image corruption and domain-shift benchmarks confirm Ramen's strong and consistent performance in complex mixed-domain scenarios.

Key takeaway

For research scientists developing or deploying vision-language models in real-world applications, Ramen offers a practical solution to maintain performance under diverse and unpredictable data shifts. You should consider integrating active sample selection and gradient caching into your test-time adaptation strategies to ensure robust model behavior across mixed domains, reducing the need for costly retraining or manual domain identification.

Key insights

Ramen robustly adapts vision-language models to mixed-domain shifts using active sample selection and an embedding-gradient cache.

Principles

• Adaptation benefits from domain-consistent samples.
• Prediction balance mitigates adaptation bias.
• Caching gradients improves adaptation efficiency.

Method

Ramen retrieves customized batches from past data using domain consistency and prediction balance criteria, then aggregates cached embedding-gradients for model updates.

In practice

• Apply active sample selection for mixed domains.
• Implement embedding-gradient caching for efficiency.

Topics

• Test-Time Adaptation
• Vision-Language Models
• Ramen Framework
• Active Sample Selection
• Embedding-Gradient Cache

Code references

• baowenxuan/Ramen

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Distill Once, Adapt Life-Long: Exploring Dataset Distillation for Continual Test-Time Adaptation — DO-ALL uses Dataset Distillation to create stable, privacy-conscious source references for robust continual test-time adaptation.
• Minibatch Selection via Partition Matroid Constrained Gradient Matching — PartitionSel optimizes LLM fine-tuning on heterogeneous data by balancing domain coverage and convergence via gradient-matching and matroid constraints.
• Teacher-Student Structure for Domain Adaptation in Ensemble Audio-Visual Video Deepfake Detection — A teacher-student framework enhances ensemble audio-visual deepfake detection models for robust generalization across diverse, unseen domains.
• Test-Time Perturbation Learning with Delayed Feedback for Vision-Language-Action Models — PDF improves VLA model robustness at test-time by mitigating overfitting and overconfidence through adaptive perturbation and delayed feedback.
• Practical transfer learning for NLP with spaCy and Prodigy — Modern NLP workflows combine pre-training, active learning, and continuous maintenance to accelerate development and improve model longevity.
• Weight Adaptation for Improving Parallel Performance of Adaptive Stochastic Natural Gradient — Adaptively adjusting weight parameters in stochastic natural gradient methods maximizes optimization improvement, especially for parallel black-box problems.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.