Profit vs. Accuracy

2026-05-18 · Source: Machine Learning on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, long

Summary

This article demonstrates a practical approach to selecting model thresholds based on expected profit and operational capacity, rather than traditional metrics like accuracy, AUC, or F1. It highlights that real-world systems operate on money, risk, and capacity constraints, especially in applications like fraud alerts, credit approvals, and insurance claims. The core idea is to maximize "Expected Profit = Benefit(TP) − Cost(FP) − Cost(Reviews) − Loss(FN)" while adhering to capacity limits, such as the number of cases analysts can review daily. The content uses a Python simulation with two hypothetical models, Model A (higher AUC) and Model B (lower AUC but better at ranking high-loss fraud), to illustrate how a model with "worse" AUC can generate higher profit under specific capacity constraints. It provides code examples for data generation, AUC calculation, profit calculation for top-K cases, and visualizations of profit versus capacity and profit versus threshold with alert volume.

Key takeaway

For Data Scientists and ML Engineers deploying models in production, you should shift your focus from solely optimizing for AUC or F1 to optimizing for profit under real-world capacity constraints. Implement profit-based thresholding by converting outcomes to monetary values and simulating profit curves. This ensures your models drive actual business value and align with operational realities, preventing queue collapse and maximizing financial returns, even if it means choosing a model with a lower AUC.

Key insights

Optimize model thresholds for profit and capacity, not just traditional accuracy metrics.

Principles

• Different error types have different costs.
• Capacity constraints change model value.
• Profit is the ultimate system metric.

Method

Simulate profit for various thresholds and capacities, then select the threshold that maximizes profit without exceeding operational limits. This involves converting outcomes to monetary values and accounting for review costs.

In practice

• Use `profit_top_k` to evaluate models under capacity.
• Visualize profit vs. capacity for leadership.
• Recalibrate thresholds periodically due to data drift.

Topics

• Profit-based Thresholding
• Expected Value Calculation
• Operational Capacity
• Fraud Detection Models
• AUC Metric

Best for: Machine Learning Engineer, Data Scientist, Director of AI/ML

Related on AIssential

• Your Model Scored 96% Accuracy. Think That’s Success? Think Again. — Model accuracy alone is insufficient; calibrated probabilities and cost-optimized thresholds drive real business value.
• A Theory of Training Profit-Optimal LLMs — Profit-optimal LLM training balances quality-driven demand with scaling costs, varying significantly by compute or data constraints.
• Your Churn Threshold Is a Pricing Decision — Churn models must optimize for dollar costs, not just accuracy, by integrating LTV and misclassification costs.
• The System Always Knows: Why Local Efficiency and System Performance Are Not the Same Problem — Optimizing local metrics like Cost per Delivery can degrade overall system performance, creating an "optimization trap" where efficiency and service diverge.
• Why Your 99% Accurate Model Might Actually Be Useless — Data leakage inflates model accuracy by providing unavailable information, leading to real-world failure.
• DIY #22 - Build a Churn Detection Model from Scratch — Customer churn prediction is a binary classification problem solvable with behavioral data and machine learning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning on Medium.