Correcting Class Imbalance in Prior-Data Fitted Networks for Tabular Classification

2026-05-23 · Source: cs.LG updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, long

Summary

Prior-data fitted networks (PFNs), particularly TabPFN-2.5, achieve exceptional performance on tabular classification but struggle with class imbalance, leading to poor accuracy for rare classes. This study investigates methods to mitigate this issue, adapting classical techniques for PFNs' in-context learning (ICL) dynamic. Experiments on 11 binary classification datasets from OpenML-CC18, using train sets with up to π₁=0.05 imbalance, revealed that PFNs are well-calibrated on balanced data but exhibit a majority class bias on imbalanced data. The research found that adjusting the decision threshold to τ=π₁ significantly improves minority class accuracy, aligning with Pₑ minimization, and yields strong balanced and worst-class accuracy. Downsampling the majority class to achieve π₀=π₁ also performs well, increasing worst-class accuracy and reducing inference computation, as TabPFN query computation scales quadratically with context size.

Key takeaway

For Machine Learning Engineers addressing class imbalance in Prior-Data Fitted Networks, adjust your decision threshold to the minority class prior probability (τ=π₁). This simple modification significantly boosts minority class accuracy without complex model changes. Alternatively, downsample the majority class to match minority samples. This improves worst-class accuracy and also reduces inference costs by decreasing context size.

Key insights

PFNs' class imbalance issues are best addressed by simple thresholding or downsampling due to their unique calibration.

Principles

• PFNs exhibit majority class bias on imbalanced data.
• Well-calibrated PFNs allow effective threshold adjustments.
• Downsampling reduces computation for ICL models.

Method

For binary classification with imbalanced PFNs, adjust the decision threshold to the minority class prior probability (τ=π₁) or downsample majority class samples to match minority class count.

In practice

• Set TabPFN decision threshold to π₁ for imbalanced data.
• Downsample majority class to equal minority samples.
• Reduce context size for faster TabPFN inference.

Topics

• Prior-Data Fitted Networks
• Tabular Classification
• Class Imbalance
• In-Context Learning
• Decision Thresholding
• Downsampling

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• When Tabular Foundation Models Meet Strategic Tabular Data: A Prior Alignment Approach — PFN-style tabular models require inference-time alignment to overcome prediction bias in strategic data environments.
• Class-Specific Branch Attention for Mitigating Gradient Interference under Class Imbalance — Mitigating inter-class gradient interference in shared representations significantly improves minority-class learning under severe imbalance.
• Learning Bayesian Network Classifiers to Minimize Class Variable Parameters — A new Bayesian network classifier minimizes class variable parameters while asymptotically estimating true probability distributions.
• Model-Agnostic Meta Learning for Class Imbalance Adaptation — HAMR dynamically addresses class imbalance and data difficulty in NLP via adaptive weighting and neighborhood-aware resampling.
• Data Presentation Over Architecture: Resampling Strategies for Credit Risk Prediction with Tabular Foundation Models — Context construction significantly impacts Tabular Foundation Model performance in imbalanced credit risk prediction.
• Does the Data Processing Inequality Reflect Practice? On the Utility of Low-Level Tasks — Low-level data processing can improve classification accuracy for practical classifiers, despite the data processing inequality.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.LG updates on arXiv.org.