Customer Churn Prediction on Structured Data Using FT-Transformer and Stacking Ensembles

2026-06-09 · Source: cs.LG updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

A new hybrid architecture for customer churn prediction on structured tabular data integrates feature-tokenized transformers (FT-Transformer) with gradient-boosted trees (XGBoost) via calibration-aware stacking. This framework addresses challenges like class imbalance using weighted loss functions and leverages out-of-fold stacking with a logistic regression meta-learner to recalibrate base model outputs. Tested on a public bank churn dataset of 10,000 customers with a 20% churn rate, the model achieved 62.10% F1, 0.861 AUC-ROC, and 0.647 PR-AUC. It statistically significantly outperformed the Multi-Layer Perceptron (MLP) baseline by 3.37 F1 points (p < 0.001) and 0.027 AUC. Ablation studies confirmed that both the transformer component and the stacking strategy materially contribute to its robust performance.

Key takeaway

For data scientists building churn prediction models, you should consider this hybrid FT-Transformer and XGBoost stacking ensemble. It significantly improves F1-score and AUC while providing well-calibrated probabilities crucial for cost-sensitive interventions. Implement class-weighted loss and out-of-fold stacking to enhance performance and ensure robust, reproducible results on imbalanced tabular datasets. This approach offers a strong balance of accuracy and interpretability for your retention strategies.

Key insights

Hybrid models combining transformers and tree ensembles offer superior, calibrated churn prediction on tabular data.

Principles

• Diverse inductive biases reduce ensemble variance.
• Stacking meta-learners improve probability calibration.
• Class-weighted loss functions preserve minority class distributions.

Method

The method involves preprocessing data, training FT-Transformer and XGBoost base models with class-weighted loss, generating out-of-fold predictions, and training a logistic regression meta-learner on these predictions.

In practice

• Use FT-Transformer for complex feature interactions.
• Employ XGBoost for discrete decision boundaries.
• Apply logistic regression as a simple, effective meta-learner.

Topics

• Customer Churn Prediction
• FT-Transformer
• XGBoost
• Stacking Ensembles
• Tabular Data
• Probability Calibration

Best for: AI Engineer, Research Scientist, Machine Learning Engineer, Data Scientist, AI Scientist

Related on AIssential

• ChurnNet: A Optimized Modern AI for Churn Prediction — Traditional ML models often outperform complex time series AI for churn prediction in performance, efficiency, and resources.
• Ensembling Tabular Foundation Models - A Diversity Ceiling And A Calibration Trap — Ensembling modern Tabular Foundation Models yields minimal gains due to high redundancy and calibration issues.
• Hybrid Imbalanced Regression Through Unified Data-Level and Algorithm-Level Balancing — The framework unifies data- and algorithm-level balancing to improve imbalanced regression performance.
• v7: Proceedings of KDD-Cup 2009 competition — The KDD-Cup 2009 Competition, held on June 28, 2009, centered on the challenge of "Fast Scoring on a Large Orange Customer Database" to predict customer behavior.
• A Universal Dense Football Event Representation Based on TabTransformer — A TabTransformer learns dense, semantic representations of heterogeneous football event data, improving downstream task performance.
• [D] Risk of using XGB models — Weak features in ensemble models can lead to overfitting and reduced robustness, despite aggregation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.LG updates on arXiv.org.