An Insight on Evaluation Metrics Under the Imbalanced Case of Anomaly Detection

2026-07-27 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

This work systematically analyzes four common anomaly detection evaluation metrics (AUROC, AUPR, F1-score, and MCC), studying their behavior across a wide range of class imbalance ratios. It introduces "metric landscapes," visualizations that relate metric values to True Positive Rate (TPR) and True Negative Rate (TNR), providing an intuitive view of metric preferences and stability. The analysis covers Time-Series Anomaly Detection (TSAD), One-Class Classification (OCC), and Image Anomaly Detection (IAD) using datasets like UCR anomaly archive, PSM, PUMP, SWaT, WaDi, CIFAR-10, MVTec AD, and ViSA. Key findings include AUROC and MCC remaining stable around 0.5 in random predictions, while AUPR and F1-score increase with positive sample proportion. The study offers practical guidance for interpreting and comparing anomaly detection results across datasets with different imbalance ratios.

Key takeaway

For Machine Learning Engineers evaluating anomaly detection models, recognize that standard metrics like AUROC, AUPR, F1-score, and MCC convey different meanings depending on the anomaly ratio. You should analyze metric landscapes to understand how these metrics emphasize True Positive Rate (TPR) and True Negative Rate (TNR) trade-offs under specific class imbalances. This ensures accurate interpretation and comparison of model performance across diverse datasets.

Key insights

Anomaly detection metric interpretation requires understanding their behavior across varying class imbalance ratios.

Principles

• Metric values depend on prediction quality and class imbalance.
• AUROC and MCC exhibit greater stability across imbalance ratios.
• AUPR and F1-score strongly emphasize positive predictions.

Method

The paper uses "metric landscapes," which map metric scores over the space of sensitivity (TPR) and specificity (TNR) operating points, to visualize metric preferences and stability under varying anomaly ratios.

In practice

• Visualize metric landscapes to understand TPR/TNR trade-offs.
• Select metrics based on anomaly ratio and class emphasis.

Topics

• Anomaly Detection
• Evaluation Metrics
• Class Imbalance
• Metric Landscapes
• AUROC
• Matthews Correlation Coefficient

Best for: Research Scientist, AI Engineer, Computer Vision Engineer, AI Scientist, Machine Learning Engineer, Data Scientist

Related on AIssential

• An Insight on Evaluation Metrics Under the Imbalanced Case of Anomaly Detection — The interpretation of AUROC, AUPR, F1-score, and MCC in anomaly detection is highly dependent on the class imbalance ratio.
• Every Classification Metric is Just Four Counts — Accuracy alone is a deceptive classification metric; all metrics derive from four confusion matrix counts.
• Rethinking Open-World Video Anomaly Detection: Diagnosing Definition Blindness — Current OWVAD evaluation suffers from "definition blindness," requiring new metrics and a definition-contrastive scoring rule for true conditional anomaly detection.
• Handling Imbalanced Classification: What Works Better Than SMOTE — SMOTE is often suboptimal for imbalanced classification; prioritize model-aware, cost-sensitive, and decision-level techniques.
• Fast and Accurate Anomaly Detection in Time Series — A novel unsupervised algorithm uses Haar discrete wavelets and a custom t-test for accurate time series anomaly detection.
• A Review of the Receiver Operating Characteristic Curve and a Proof About the Area Beneath It — The area under an ROC curve quantifies a binary classifier's discriminative power under a specific non-overlap hypothesis.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.