Tuning CatBoost: the hierarchy of needs (and why depth comes before learning rate)

2026-05-22 · Source: Valeriy’s Substack · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, quick

Summary

This article, published on May 22, 2026, by Valeriy Manokhin, critiques the common approach to CatBoost hyperparameter tuning found in many online guides. It identifies a prevalent issue where resources merely list 20 hyperparameters alphabetically, lacking a structured methodology for optimization. The author proposes a "hierarchy of needs" for CatBoost tuning, specifically asserting that the "depth" hyperparameter should be prioritized and adjusted before the "learning rate" to achieve more effective and efficient model performance.

Key takeaway

For Machine Learning Engineers or Data Scientists struggling with CatBoost optimization, this article suggests abandoning generic alphabetical hyperparameter lists. You should adopt a structured tuning strategy, specifically prioritizing the "depth" parameter's adjustment before moving to the "learning rate" to achieve more efficient and effective model performance. This approach can streamline your tuning process and yield better predictive models.

Key insights

CatBoost tuning requires a structured approach, prioritizing depth before learning rate for optimal results.

Principles

• Hyperparameter tuning benefits from a hierarchical approach.
• CatBoost's "depth" parameter precedes "learning rate" in tuning priority.

Method

Prioritize CatBoost's "depth" hyperparameter adjustment before tuning the "learning rate" for effective optimization.

Topics

• CatBoost
• Hyperparameter Tuning
• Gradient Boosting
• Tree Depth
• Learning Rate
• Model Optimization

Best for: Machine Learning Engineer, Data Scientist

Related on AIssential

• Where CatBoost beats XGBoost and LightGBM — and what the book is honest about — CatBoost leads GBDT rankings on diverse tabular datasets, excelling with categorical features due to its unique architectural decisions.
• Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration — Complete(d)P enables effective per-module hyperparameter transfer across diverse scaling axes for large models.
• Top 20 CatBoost Interview Questions and Answers (Part 2 of 2) — CatBoost is a gradient boosting algorithm designed for high accuracy with minimal preprocessing, handling categorical features and overfitting robustly.
• Foundations of Deep Learning: Gradient Descent, Back Propagation and Self-Attention — Gradient Descent, Back Propagation, and Self-Attention are core algorithms enabling modern neural networks and LLMs.
• [P] PerpetualBooster v1.9.0 - GBM with no hyperparameter tuning, now with built-in causal ML, drift detection, and conformal prediction — PerpetualBooster simplifies GBM deployment by replacing hyperparameter tuning with a single budget parameter.
• 7 XGBoost Tricks for More Accurate Predictive Models — Optimizing XGBoost models for accuracy involves strategic hyperparameter tuning and regularization techniques.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Valeriy’s Substack.