When One Tree Grows Too Deep: A Decision Tree Case Study on Customer Churn

2026-07-26 · Source: Data Science on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, medium

Summary

A case study on the Kaggle Telco Customer Churn dataset demonstrates the pitfalls of an unrestricted Decision Tree model. While Decision Trees offer high interpretability by using Gini Impurity to make sequential yes/no decisions, allowing one to grow to a maximum depth of 27 resulted in significant overfitting. This unconstrained model achieved an accuracy of 0.7036 and an F1-Score of 0.4688, underperforming a simpler Logistic Regression model which scored 0.79 accuracy and 0.56 F1-Score. The root split identified "tenure" as the most predictive feature with a Gini Index of 0.3903. The analysis revealed that an overfit tree memorizes training data noise, leading to a low Precision of 0.4477, indicating a high rate of false churn predictions and wasted resources. This highlights the necessity of regularization techniques or ensemble methods like Random Forest to prevent such issues.

Key takeaway

For data scientists building classification models, if you are considering Decision Trees for their interpretability, you must explicitly constrain their growth. Allowing unrestricted depth, as shown by the 27-level tree, leads to severe overfitting and poor generalization, resulting in wasted business resources from false positives. Instead, implement "max_depth" limits or "min_samples_leaf" to ensure your model learns generalizable patterns, or opt for ensemble methods like Random Forest for more robust production-ready performance.

Key insights

Unrestricted Decision Trees overfit by memorizing training data noise, leading to poor generalization despite interpretability.

Principles

• Decision Trees use Gini Impurity to find optimal splits.
• Unrestricted tree depth causes severe overfitting.
• Root split identifies the most predictive feature.

Method

Decision Trees recursively split nodes based on Gini Impurity reduction, aiming for pure child nodes. This greedy process continues until data is exhausted or a class is pure.

In practice

• Set "max_depth" to prevent overfitting.
• Implement "min_samples_split" or "min_samples_leaf".
• Use Decision Trees for quick feature diagnostics.

Topics

• Decision Trees
• Customer Churn Prediction
• Model Overfitting
• Gini Impurity
• Ensemble Methods
• Model Interpretability

Best for: Machine Learning Engineer, Data Scientist, AI Student

Related on AIssential

• Decision Trees Demystified: The Algorithm That Thinks Like You Do — Decision trees mimic human decision-making by optimally reducing uncertainty through recursive data splits, offering high interpretability.
• DIY #22 - Build a Churn Detection Model from Scratch — Customer churn prediction is a binary classification problem solvable with behavioral data and machine learning.
• From Decision Trees to Forests: Making Logical Choices in Machine Learning — Decision trees use data splitting and ensemble methods to learn patterns, avoiding overfitting through careful validation and diverse metrics.
• Ternary Decision Trees with Locally-Adaptive Uncertainty Zones — Ternary decision trees improve accuracy by explicitly modeling and blending predictions for boundary-uncertain instances.
• Hinge Regression Tree: A Newton Method for Oblique Regression Tree Splitting — Hinge Regression Trees use a damped Newton method to learn powerful oblique splits, achieving universal approximation.
• Trees Knew Where the Model Was Weak. Now a Mixture of Experts Repairs It. — A model's internal "leaf kernel" can map its weaknesses, enabling targeted repair with monotone Mixture of Experts while preserving interpretability.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Data Science on Medium.