Quality Text, Robust Vision: The Role of Language in Enhancing Visual Robustness of Vision-Language Models

2026-07-27 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy · Depth: Expert, extended

Summary

Quality Text-guided Adversarial Fine-Tuning (QT-AFT) is a novel method enhancing the visual robustness of pre-trained Vision-Language Models (VLMs) like CLIP against adversarial attacks. Existing adversarial training (AT) approaches either overfit to class labels (supervised methods like TeCoA) or lack semantic awareness (unsupervised methods like FARE), limiting zero-shot robustness across diverse tasks. QT-AFT addresses this by using high-quality, detailed image captions, generated by models such as InternVL-2.5-8B, to guide adversarial example generation. This approach encourages the visual encoder to robustly recognize a broader range of image features under adversarial noise. Evaluated on CLIP-ViT-B/16 trained on ImageNet and tested across 16 zero-shot datasets against AutoAttack (ϵ=4/255), QT-AFT achieved leading zero-shot adversarial robustness on 12 datasets and the best average performance, while also maintaining high clean accuracy. Further analysis revealed that linguistic elements beyond object names, such as adjectives, adverbs, and function words, significantly contribute to robustness.

Key takeaway

For Machine Learning Engineers developing robust Vision-Language Models, you should integrate high-quality, detailed image captions into your adversarial fine-tuning processes. Relying solely on class labels or unsupervised methods limits zero-shot robustness and clean accuracy. By using rich linguistic supervision, including adjectives, adverbs, and proper word order, you can significantly improve model resilience across diverse unseen tasks. This approach ensures your VLMs maintain performance even under adversarial conditions.

Key insights

Leveraging high-quality, semantically rich linguistic supervision via detailed captions significantly enhances zero-shot adversarial robustness in vision-language models.

Principles

• Detailed linguistic supervision is crucial for VLM zero-shot adversarial robustness.
• Overfitting to class labels limits robustness generalization.
• Adjectives, adverbs, and function words improve transferable robustness.

Method

QT-AFT generates adversarial examples by combining an unsupervised objective with a caption-guided contrastive loss, using synthetically generated detailed captions (e.g., ~50 words from InternVL-2.5-8B).

In practice

• Employ detailed image captions for VLM adversarial fine-tuning.
• Emphasize descriptive language in captions for broader robustness.
• Adapt caption content based on task, like removing nouns for textures.

Topics

• Vision-Language Models
• Adversarial Robustness
• Zero-shot Learning
• Adversarial Training
• Image Captioning
• CLIP

Code references

• cvlab-columbia/ZSRobust4FoundationModel
• serendipity1122/Pre-trained-Model-Guided-Fine-Tuning-for-Zero-Shot-Adversarial-Robustness
• zhyblue424/TGA-ZSR
• chs20/RobustVLM

Best for: Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Security Engineer

Related on AIssential

• Dual Adversarial Fine-tuning for Enhancing Robustness of Large Vision Language Model — A dual adversarial fine-tuning framework enhances LVLM robustness and generalizability by jointly optimizing visual and semantic signals.
• Attributes Should Come from Images, Not Class Names: Distribution-Conditioned Attribute Selection for Vision-Language Models — LLM-generated class attributes are label-conditioned; image-derived attributes improve VLM zero-shot classification and robustness.
• Zero-Shot Captioning for Cultural Heritage: Automated Image Analysis of Traditional Indonesian Clothing — Retrieval-augmented vision-language models effectively generate culturally accurate captions for low-resource heritage data.
• Towards Robustness against Typographic Attack with Training-free Concept Localization — CLIP models' vulnerability to typographic attacks stems from specific ViT components encoding lexical information.
• Diagnosing Visual Ignorance in Vision-Language Models — Vision-Language Models frequently prioritize language priors over visual evidence due to internal routing failures and flawed benchmark evaluations.
• The Hardest Part of Fine-Tuning Isn’t the Training — The most challenging aspect of NLP fine-tuning is effective label design and meticulous data annotation, not the model training itself.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.