Can perceptual similarity metrics be used to compare adversarial attacks?

2023-12-07 · Source: Blog of the TransferLab — appliedAI Institute · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision, Adversarial Machine Learning · Depth: Advanced, extended

Summary

This analysis investigates the suitability of perceptual similarity metrics, specifically LPIPS (Learned Perceptual Image Patch Similarity), for comparing adversarial attacks in computer vision. Adversarial samples are crafted by perturbing images to cause misclassification by a model while remaining imperceptible to humans. The study highlights that traditional $\ell_p$ norms are inadequate for comparing diverse attack types, leading to the exploration of Wasserstein-based attacks and perceptual metrics like SSIM, FSIM, and LPIPS. Experiments using the Imagenette dataset, a VGG-11 architecture, and LPIPS with VGG-16, AlexNet, and SqueezeNet backends demonstrate that LPIPS itself is susceptible to adversarial perturbations. While PGD attacks with $\ell_{\infty}$ norm showed limited transferability to LPIPS with different backends, Wasserstein attacks exhibited high transferability, rendering LPIPS unsuitable for comparing these attack methods.

Key takeaway

For AI Scientists and Research Scientists evaluating adversarial robustness, relying solely on LPIPS or similar perceptual metrics for comparing diverse adversarial attacks is problematic. Your evaluations should incorporate human perceptual judgment, as LPIPS is shown to be susceptible to adversarial perturbations, especially from Wasserstein-based attacks. This necessitates continued investment in human-in-the-loop validation or the development of more robust, perceptually aligned metrics.

Key insights

Perceptual similarity metrics like LPIPS are vulnerable to adversarial attacks, limiting their use for comparing different attack types.

Principles

• $\ell_p$ norms are insufficient for comparing diverse adversarial attacks.
• Adversarial attacks can transfer across different neural network architectures.
• Human judgment remains critical for evaluating adversarial imperceptibility.

Method

The study compares LPIPS score distributions between images perturbed by adversarial attacks and "fake adversaries" (randomly perturbed images) to assess metric susceptibility.

In practice

• Use human surveys for robust adversarial attack evaluation.
• Consider Wasserstein attacks for higher transferability.
• Be aware of LPIPS's vulnerability to adversarial manipulation.

Topics

• Adversarial Attacks
• Perceptual Metrics
• LPIPS
• Wasserstein Distance
• Computer Vision Robustness

Code references

• bezirganyan/adversarial_arena
• fastai/imagenette

Best for: AI Scientist, Research Scientist, AI Researcher, Deep Learning Engineer, Computer Vision Engineer

Related on AIssential

• Generalised Eigenvalue Geometry of Semantic Adversarial Attacks — Semantic adversarial vulnerability is governed by the relative local geometry of proxy and target embedding models.
• Architectural Bias in Face Presentation Attack Detection: A Comparative Study of Vision Transformers and Convolutional Neural Networks — Pretrained Vision Transformers significantly reduce demographic bias and improve generalization in face Presentation Attack Detection.
• Robustness of Vision Foundation Models to Common Perturbations — Vision foundation models are non-robust to common image perturbations, impacting downstream task performance.
• Semantic-aware Adversarial Fine-tuning for CLIP — Semantically enriched text descriptions improve CLIP's adversarial robustness by generating more effective adversarial examples.
• Quality-Preserving Imperceptible Adversarial Attack on Skeleton-based Human Action Recognition — Adversarial attacks on S-HAR can be imperceptible by preserving motion quality through distribution-based optimization.
• Adversarial Flow Matching for Imperceptible Attacks on End-to-End Autonomous Driving — AFM leverages Flow Matching to create imperceptible, transferable gray-box attacks against Transformer-based autonomous driving systems.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Blog of the TransferLab — appliedAI Institute.