Robustness of Vision Foundation Models to Common Perturbations

2026-04-16 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision & Pattern Recognition, Cybersecurity & Data Privacy · Depth: Expert, quick

Summary

A systematic study evaluates the robustness of vision foundation models to common image perturbations like JPEG compression, brightness, and contrast adjustments. The research introduces three new robustness metrics and defines five mathematical properties for these metrics, analyzing their adherence. Six industry-scale foundation models from OpenAI and Meta were assessed across nine perturbation categories, revealing a general lack of robustness. The study also demonstrates that these perturbations degrade downstream application performance, such as classification accuracy, and that the proposed robustness metrics can predict these performance impacts. Additionally, a fine-tuning approach is presented to enhance model robustness without compromising utility.

Key takeaway

For research scientists developing or deploying vision foundation models, you should prioritize evaluating your models against common image perturbations. The study indicates that these models are generally non-robust, and this lack of robustness directly impacts downstream task performance. Consider implementing the proposed fine-tuning approach to enhance model resilience without sacrificing overall utility, ensuring more reliable real-world application.

Key insights

Vision foundation models are non-robust to common image perturbations, impacting downstream task performance.

Principles

• Common perturbations alter image embedding vectors.
• Robustness metrics can predict performance degradation.

Method

The study proposes three robustness metrics with five mathematical properties, then evaluates six industry-scale vision foundation models across nine perturbation categories, and finally suggests a fine-tuning approach.

In practice

• Evaluate models against common image edits.
• Fine-tune models to improve perturbation robustness.

Topics

• Vision Foundation Models
• Image Perturbations
• Model Robustness
• Robustness Metrics
• Downstream Task Performance

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Harnessing non-adversarial robustness in large language models — LLM robustness to prompt variations can be efficiently acquired via fine-tuning, avoiding full retraining.
• Unlocking the Power of Critical Factors for 3D Visual Geometry Estimation — Data diversity, specific loss functions, and joint alignment are critical for visual geometry estimation performance.
• Physically-Induced Atmospheric Adversarial Perturbations: Enhancing Transferability and Robustness in Remote Sensing Image Classification — FogFool generates physically plausible, fog-based adversarial perturbations for robust remote sensing image classification attacks.
• Scaling Capability in Token Space: An Analysis of Large Vision Language Model — LVLMs exhibit predictable sublinear and linear scaling regimes based on vision token count.
• BadWorld: Adversarial Attacks on World Models — BadWorld reveals severe structural fragility in Visual World Models through novel self-supervised and trajectory-adaptive adversarial attacks.
• On the Reliability of Cue Conflict and Beyond — Flawed cue-conflict benchmarks yield unstable neural network shape-texture bias estimates; REFINED-BIAS offers a reliable, interpretable diagnostic framework.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.