VendorBench-100: A Unified Cross-Paradigm Benchmark for Deepfake Image Detection

2026-07-07 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy · Depth: Expert, quick

Summary

VendorBench-100 is a new cross-paradigm benchmark designed to evaluate deepfake image detection across commercial APIs, zero-shot vision-language models (LLMs), and open-source detectors. It uses a single adversarial 100-image corpus, a unified output schema, and a common evaluation framework to assess 36 representative models. The benchmark prioritizes challenging real-world scenarios through a curated taxonomy of eight edge-case families, including face swaps and AI photo edits, and ranks models primarily by Matthews correlation coefficient (MCC), also reporting ROC-AUC. Evaluation results indicate commercial APIs generally achieve the strongest median performance, followed by vision LLMs and open-source detectors, though some open-source models are competitive. A key finding is the consistent divergence between ROC-AUC and MCC, showing that strong score discrimination does not guarantee reliable default-threshold decisions. The complete framework and results are publicly released.

Key takeaway

For AI Security Engineers evaluating deepfake detection solutions, you should prioritize benchmarks that assess performance across diverse paradigms and real-world edge cases. Do not solely rely on ROC-AUC; instead, scrutinize metrics like MCC, especially with imbalanced data, to ensure reliable operating-point decisions. Your choice of detector should consider its performance on specific edge-case families and its default threshold reliability, not just its general discriminative power.

Key insights

VendorBench-100 unifies deepfake detection evaluation, revealing commercial APIs lead but metric disagreement is critical.

Principles

• Cross-paradigm evaluation is crucial.
• MCC better assesses imbalanced data.
• ROC-AUC doesn't imply reliable thresholds.

Method

VendorBench-100 evaluates 36 models using a 100-image adversarial corpus, unified output schema, and common framework, ranking primarily by MCC and reporting ROC-AUC.

In practice

• Use VendorBench-100 for detector comparison.
• Prioritize MCC for imbalanced datasets.
• Scrutinize default operating points.

Topics

• Deepfake Detection
• Image Forensics
• Vision-Language Models
• Commercial APIs
• Benchmark Datasets
• Adversarial Robustness
• Evaluation Metrics

Code references

• sharayu-20/vendorbench-100

Best for: AI Engineer, Machine Learning Engineer, Research Scientist, AI Scientist, Computer Vision Engineer, AI Security Engineer

Related on AIssential

• How Researchers Measure, Detect and Benchmark AI Manipulation — Deepfake technology lacks a universal definition, necessitating standardized metrics and diverse datasets for robust detection and generation.
• SwordBench: Evaluating Orthogonality of Steering Image Representations — SwordBench evaluates image representation steering, introducing metrics for cross-concept robustness and collateral damage.
• Detecting Deepfakes with Multivariate Soft Blending and CLIP-based Image-Text Alignment — Multimodal image-text alignment and blended data augmentation significantly improve deepfake detection generalization and robustness.
• WorldBench: A Challenging and Visually Diverse Multimodal Reasoning Benchmark — Visual diversity is crucial for robust multimodal reasoning benchmark development.
• Architecture-Adaptive Uncertainty Fusion for Deepfake Detection — Correlation-Optimized Fusion adaptively combines five uncertainty sources to improve deepfake detection reliability, especially under distribution shift.
• Small Model, Big Results: How a 5MB On-Device NSFW Detector Outperforms Cloud APIs and Passes a… — Small, on-device object detection models can surpass large cloud APIs in NSFW accuracy and fairness by focusing on anatomical shapes.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.