Beyond Cross-Validation: Adaptive Parameter Selection for Kernel-Based Gradient Descents

2026-03-05 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

This paper proposes a novel Hybrid Selection Strategy (HSS) for Kernel-Based Gradient Descent (KGD) algorithms, which integrates bias-variance analysis with the splitting method for adaptive parameter selection. The strategy introduces the concept of empirical effective dimension to quantify iteration increments, theoretically demonstrating that KGD equipped with HSS achieves optimal generalization error bounds and adapts effectively to different kernels, target functions, and error metrics. Numerical experiments show HSS's superior performance in accuracy and efficiency compared to existing methods like hold-out, AIC, BIC, and Lepskii principle, especially under the "$L_{\infty}$ norm". Additionally, HSS effectively addresses the covariate shift problem, validating its theoretical optimal generalization error bound guarantee under the "$L_{\infty}$ metric". The research suggests future applications in distributed learning and adaptive parameter selection for spherical data.

Key takeaway

A novel Hybrid Selection Strategy (HSS) for Kernel-Based Gradient Descent (KGD) adaptively determines optimal iterations by integrating bias-variance analysis with a small-subset splitting method. This approach achieves optimal generalization error bounds, adapting simultaneously to target function regularity ($r \in [1/2, \infty)$), kernel capacity ($s \in (0,1]$), and multiple error metrics (e.g., $L_2, L_{\infty}$). HSS significantly outperforms traditional methods like Hold-Out in $L_{\infty}$ accuracy and robustness to covariate shift, offering a computationally efficient and theoretically optimal solution for precise KGD parameter tuning.

Topics

• Kernel-Based Gradient Descent
• Hyperparameter Selection
• Bias-Variance Analysis
• Generalization Error Bounds
• Covariate Shift

Code references

• Ariesoomoon/HSS_experiments

Best for: AI Researcher, AI Scientist, Research Scientist

Related on AIssential

• AdaGrad-Diff: A New Version of the Adaptive Gradient Algorithm — AdaGrad-Diff improves optimization robustness by adapting stepsizes based on gradient differences, not just gradient magnitudes.
• Kernel Selection is Model Selection: A Unified Complexity-Penalized Approach for MMD Two-Sample Tests — CP-MMD unifies kernel selection as a complexity-penalized model selection problem for robust two-sample testing.
• Automated Kernel Discovery Towards Understanding High-dimensional Bayesian Optimization — LLM-driven kernel discovery via a two-stage process and LOO-CRPS significantly improves high-dimensional Bayesian optimization.
• Large-scale empirical tuning and comparison of default optimizers for variational inference — A selection of 5 adaptive stochastic optimizers reliably approximates optimal performance for black-box variational inference without extensive tuning.
• Minibatch Selection via Partition Matroid Constrained Gradient Matching — PartitionSel optimizes LLM fine-tuning on heterogeneous data by balancing domain coverage and convergence via gradient-matching and matroid constraints.
• Stochastic Gradient Methods: Bias, Stability and Generalization — A new framework analyzes stability and generalization for biased stochastic gradient methods.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.