From Chosen to Learned Kernels

2026-01-11 · Source: Agus’s Substack · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, medium

Summary

"From Chosen to Learned Kernels," Chapter 3 of "The Learned Kernel," challenges the practice of manually selecting kernel bandwidths, arguing they are critical parameters, not mere hyperparameters. The article demonstrates that a single bandwidth choice can drastically impact model performance, showing a \$55k RMSE difference on California housing data (e.g., \$118k vs. \$63k). It introduces the Automatic Relevance Determination (ARD) kernel, k_θ(x, x') = exp(−½ Σⱼ₌₁ᵈ (x_j − x'_j)² / ℓ_j²), which learns a unique length scale ℓ_j for each feature, achieving a test RMSE of 0.592 compared to 0.639 for an optimally chosen isotropic bandwidth. The analysis identifies two key hazards in learning kernels: the non-identifiability of the kernel's overall scale when combined with ridge regularization (λ), resolved by constraining the kernel's diagonal to one (k_θ(x, x) ≡ 1), and overfitting the geometry itself, which requires fitting the kernel on one data split and evaluating it on another to prevent data leakage.

Key takeaway

For Machine Learning Engineers optimizing kernel-based models, stop manually tuning kernel bandwidths; instead, treat them as learnable parameters. You should implement ARD kernels, constraining the diagonal to one (k_θ(x, x) ≡ 1) to ensure scale identifiability. Always fit your kernel's geometry on one data split and evaluate it on another to prevent overfitting the geometry itself, ensuring your model learns true data structure.

Key insights

Kernel geometry, critical for model performance, must be learned from data, not manually chosen, while addressing identifiability and leakage.

Principles

• Parameters significantly affecting performance must be learned.
• Constrain kernel diagonal to one (k_θ(x, x) ≡ 1) for scale identifiability.
• Overfitting the geometry requires separate data splits for fitting and evaluation.

Method

Employ an ARD kernel, k_θ(x, x') = exp(−½ Σⱼ₌₁ᵈ (x_j − x'_j)² / ℓ_j²), to learn per-feature length scales ℓ_j by minimizing error on a held-out fold.

In practice

• ARD kernels can reduce test RMSE (e.g., 0.592 vs. 0.639).
• Feature relevance (1/ℓ_j) indicates local sensitivity, not overall importance.

Topics

• Kernel Methods
• Automatic Relevance Determination
• Hyperparameter Learning
• Model Overfitting
• Data Leakage
• Ridge Regularization

Code references

• asudjianto-xml/Learned-Kernel

Best for: Machine Learning Engineer, AI Scientist, Data Scientist

Related on AIssential

• The Kernel as the Normal Form — Most machine learning methods are variations of a kernel machine, with geometry (the kernel) as the primary choice.
• Kernel-based guarantees for nonlinear parametric models in Bayesian optimization — A kernel-based framework provides theoretical guarantees for nonlinear parametric models in adaptive optimization.
• Data Leakage in Machine Learning: Why You Must Split Before Preprocessing — Preprocessing steps that learn from data must occur after splitting to prevent data leakage and ensure valid model evaluation.
• Kernel Selection is Model Selection: A Unified Complexity-Penalized Approach for MMD Two-Sample Tests — CP-MMD unifies kernel selection as a complexity-penalized model selection problem for robust two-sample testing.
• Why Every Supervised Model You’ve Ever Trained Has a Geometric Blind Spot (And Why More Data Won’t… — Supervised models have an inherent geometric blind spot due to their objective function, causing structural failures.
• On the Robustness of Kernel Goodness-of-Fit Tests — Existing kernel goodness-of-fit tests lack robustness, necessitating a new framework for practical model validation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Agus’s Substack.