The Hidden Self-Attention Inside Gradient-Boosted Trees

2026-01-11 · Source: Agus’s Substack · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, long

Summary

A new framework unifies Gradient-Boosted Trees (GBDTs), kernel regression, and self-attention under a "Generalized Nadaraya–Watson (GNW) operator," revealing a shared structure: models learn a similarity geometry, attach values to neighborhoods, and aggregate them for prediction. GBDTs implicitly learn a task-adaptive, supervised kernel through leaf co-membership, defining proximity based on shared leaves across trees. The GNW operator decomposes prediction into query (Q), key (K), value (V), and an optional decoder (D). This analysis demonstrates a fitted GBDT is an exact QKV attention head, with each tree acting as a discrete, hard-routed attention mechanism. The framework allows independent diagnosis and improvement of geometry, values, or the decoder. For instance, replacing GBDT's learned leaf scores with raw labels underperforms, while Kernel Ridge Regression (KRR) or neural value heads on the GBDT geometry can be competitive. A diagnostic method for selecting value heads achieved performance within 5% of the empirical best on 73% of dataset-seed configurations across ten OpenML benchmarks.

Key takeaway

For Machine Learning Engineers optimizing GBDT performance on tabular data, you should view GBDTs as learned QKV attention heads. This allows you to systematically improve models by diagnosing and upgrading specific components: the learned geometry, the assigned values, or the final decoder. Instead of replacing GBDTs entirely, consider extracting the GBDT's strong supervised geometry and then refitting more expressive value heads, like Kernel Ridge Regression or neural networks, based on diagnostic signals. This targeted approach can enhance performance without discarding the GBDT's inherent strengths.

Key insights

GBDTs, kernel regression, and self-attention share a QKV operator structure, differing in where they place complexity.

Principles

• Learning involves geometry discovery and value assignment.
• Model performance depends on geometry, values, and decoder.
• GBDTs' geometry is front-loaded, Transformers' nonlinearity is back-loaded.

Method

The Generalized Nadaraya–Watson (GNW) operator decomposes prediction into Q(x), Kᵢ, κ, Vᵢ, and D. Weights wᵢ(x) = κ(Q(x), Kᵢ) / ∑ⱼ κ(Q(x), Kⱼ) aggregate values Vᵢ, followed by decoder D.

In practice

• Extract GBDT leaf-membership as a fixed supervised embedding.
• Refit ridge or neural heads on leaf features to improve values.
• Use diagnostics to identify geometry, value, or decoder bottlenecks.

Topics

• Gradient Boosting
• Decision Trees
• Self-Attention
• Kernel Regression
• Nadaraya-Watson Operator
• Tabular Data Modeling
• Model Diagnostics

Code references

• asudjianto-xml/geomlearn

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• The Model Already Knows Where It’s Wrong — A GBDT's internal geometry reveals local weaknesses, enabling diagnosis-specific model repair beyond global metrics.
• Attention Is Not What You Think! — In-context learning for tabular data is row attention, a mechanism Random Forests have employed for 20 years.
• Gating Enables Curvature: A Geometric Expressivity Gap in Attention — Multiplicative gating in attention enables non-flat geometric representations, enhancing expressivity for nonlinear tasks.
• Gradient Regularized Newton Boosting Trees with Global Convergence — A gradient-regularized Newton boosting scheme achieves global convergence for GBDTs, addressing prior theoretical gaps.
• PLS in the Mirror of Self-Attention — PLS can be cast as linearized self-attention, suggesting self-attention's dimensionality normalization improves learning.
• [R] Analysis of 350+ ML competitions in 2025 — ML competition trends show shifts towards foundation models and increased compute, while GBDTs and PyTorch maintain strong positions.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Agus’s Substack.