VendorBench-100: A Unified Cross-Paradigm Benchmark for Deepfake Image Detection

2024-02-04 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy, Data Science & & Analytics · Depth: Expert, extended

Summary

The VendorBench-100 benchmark evaluates 36 deepfake image detection models across three paradigms: 5 commercial APIs, 7 zero-shot vision-language models (LLMs), and 24 open-source detectors. It uses a unified protocol on a challenging, adversarial 100-image corpus (79 fake, 21 real) featuring eight edge-case families, including face swaps, text-to-video stills, and AI photo edits. Models are primarily ranked by Matthews correlation coefficient (MCC), with ROC-AUC as a tie-breaker. Commercial APIs generally show the strongest median performance, followed by vision LLMs, and then open-source detectors. A key finding is the consistent divergence between ranking ability (ROC-AUC) and operating-point quality (MCC), indicating that strong score discrimination does not guarantee reliable default-threshold decisions. The evaluation framework and results are open-sourced for reproducibility.

Key takeaway

For AI Scientists and Machine Learning Engineers evaluating deepfake detection solutions, you must assess models using both Matthews correlation coefficient (MCC) and ROC-AUC. Relying solely on one metric, like accuracy or F1, can mask critical flaws in a model's default decision threshold, even if its underlying discriminative ability is strong. Your procurement or deployment decisions should explicitly weigh both operating-point quality and ranking power to avoid unreliable systems that appear strong on partial metrics.

Key insights

Deepfake detection performance varies significantly across paradigms, with metric divergence highlighting unreliable default decision thresholds.

Principles

• Commercial APIs lead in median performance.
• ROC-AUC and MCC measure distinct detector qualities.
• Class imbalance invalidates raw accuracy/F1 scores.

Method

VendorBench-100 uses a 100-image adversarial corpus with 8 edge-case families, a unified output schema, and dual-metric evaluation (MCC primary, ROC-AUC tiebreak) for 36 models.

In practice

• Use MCC and ROC-AUC for detector evaluation.
• Prioritize commercial APIs for typical performance.
• Investigate DRCT for strong open-source ranking.

Topics

• Deepfake Detection
• AI-Generated Image Detection
• Cross-Paradigm Benchmarking
• Matthews Correlation Coefficient
• ROC-AUC
• Vision-Language Models
• Commercial APIs

Code references

• sharayu-20/vendorbench-100
• bit-ml/DeCLIP
• beibuwandeluori/DRCT
• mever-team/rine
• Michel-liu/FatFormer

Best for: AI Engineer, Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, Director of AI/ML

Related on AIssential

• VendorBench-100: A Unified Cross-Paradigm Benchmark for Deepfake Image Detection — VendorBench-100 unifies deepfake detection evaluation, revealing commercial APIs lead but metric disagreement is critical.
• Detecting Deepfakes with Multivariate Soft Blending and CLIP-based Image-Text Alignment — Multimodal image-text alignment and blended data augmentation significantly improve deepfake detection generalization and robustness.
• How Researchers Measure, Detect and Benchmark AI Manipulation — Deepfake technology lacks a universal definition, necessitating standardized metrics and diverse datasets for robust detection and generation.
• Architecture-Adaptive Uncertainty Fusion for Deepfake Detection — Correlation-Optimized Fusion adaptively combines five uncertainty sources to improve deepfake detection reliability, especially under distribution shift.
• SwordBench: Evaluating Orthogonality of Steering Image Representations — SwordBench evaluates image representation steering, introducing metrics for cross-concept robustness and collateral damage.
• Training-Free Metrics for Synthetic Object Detection Data: A Proxy for Detector Performance — CCDM metrics provide a training-free, highly accurate proxy for evaluating synthetic object detection data utility, outperforming existing methods.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.