Robustifying Vision-Language Models via Test-Time Prompt Adaptation

2026-07-10 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, quick

Summary

A new framework named RITA (Robust test-tIme prompt-TAdaptation) significantly enhances the adversarial robustness of pre-trained Vision-Language Models (VLMs) like CLIP. While VLMs exhibit strong zero-shot generalization, their performance sharply declines when faced with adversarial perturbations. Current test-time adaptation methods, which rely on sample-level confidence heuristics, often fail to distinguish confident adversarial mispredictions from true semantic consistency. RITA addresses this by shifting from sample-level estimates to distribution-level alignment, leveraging the observation that adversarial distortion, despite corrupting holistic representations, often preserves semantic integrity within the distribution of augmented views. The framework employs optimal transport to align augmented visual features with textual prototypes, thereby mitigating adversarial outliers and correcting cross-modal semantic misalignment. Additionally, RITA incorporates a dynamic cache to accumulate reliable cues from the test stream for continuous online refinement, demonstrating improved adversarial robustness without sacrificing clean accuracy.

Key takeaway

For Computer Vision Engineers developing robust Vision-Language Models, RITA offers a critical shift from sample-level to distribution-level adaptation. You should consider integrating optimal transport for aligning augmented visual features with textual prototypes to mitigate adversarial outliers. This approach significantly improves adversarial robustness without compromising clean accuracy, providing a more reliable defense against sophisticated attacks in real-world deployments. Implement a dynamic cache to refine your models continuously with reliable test stream cues.

Key insights

Adversarial attacks corrupt VLM representations, but semantic integrity can be preserved via distribution-level alignment.

Principles

• Adversarial distortion is structurally brittle.
• Distribution-level alignment enhances robustness.
• Online refinement improves adaptation.

Method

RITA uses optimal transport to align augmented visual features with textual prototypes and employs a dynamic cache for online refinement from test stream cues.

In practice

• Apply optimal transport for feature alignment.
• Implement dynamic caching for continuous adaptation.
• Use augmented views to preserve semantic integrity.

Topics

• Vision-Language Models
• Adversarial Robustness
• Test-Time Adaptation
• Prompt Adaptation
• Optimal Transport
• CLIP

Best for: Research Scientist, AI Scientist, Computer Vision Engineer

Related on AIssential

• SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models — SS-TPT enhances VLM adversarial robustness by dynamically weighting augmented views using stability and suitability scores, optimizing robustness-throughput.
• Rethinking Layer-Wise Information Allocation for Vision Foundation Model Adaptation — Effective prompt-based VFM adaptation requires principled layer-wise information allocation, not just increased prompt capacity.
• On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMs — RL-finetuned VLMs face an accuracy-faithfulness trade-off, compromising robustness and CoT consistency.
• GEASS: Gated Evidence-Adaptive Selective Caption Trust for Vision-Language Models — Naively using VLM-generated captions can degrade accuracy due to anchoring effects and asymmetric error types.
• Harnessing non-adversarial robustness in large language models — LLM robustness to prompt variations can be efficiently acquired via fine-tuning, avoiding full retraining.
• On Adversarial Vulnerability of Vision-Language Models through the Lens of Intermediate Spectral Subspaces — Adversarial vulnerability in vision-language models stems from the spectral structure of intermediate linear transformations.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.