Automated Kernel Discovery Towards Understanding High-dimensional Bayesian Optimization

2026-02-09 · Source: cs.LG updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

Kernel Discovery, an LLM-driven evolutionary framework, addresses the challenge of designing effective Gaussian Process (GP) kernels for high-dimensional Bayesian optimization (BO). It overcomes limitations of existing automated approaches by employing a two-stage process: an LLM first proposes novel mathematical kernel forms, and a second LLM converts these into executable code. The framework introduces Leave-one-out Continuous Ranked Probability Score (LOO-CRPS) as a selection criterion to penalize overfitted kernels. Evaluated on five high-dimensional BO benchmarks (Rover, Mopta08, Lasso-DNA, SVM, Humanoid, with dimensions from D=100 to D=6392), Kernel Discovery achieved an average rank of 1.2 out of 17 methods, significantly outperforming competitive baselines like Compositional Search (rank 4.4) and other LLM-based BO methods. Analysis revealed that compositions of geometrically-warped and non-stationary kernels contribute to performance improvements.

Key takeaway

For Machine Learning Engineers optimizing high-dimensional black-box functions, you should consider integrating LLM-driven kernel discovery into your Bayesian optimization pipelines. This approach, particularly its two-stage mathematical formulation and code generation, can yield more effective and diverse Gaussian Process kernels than traditional methods. By adopting LOO-CRPS for kernel selection, you can mitigate overfitting and achieve superior performance, as demonstrated by an average rank of 1.2 on challenging benchmarks. Explore novel kernel structures, including geometric warpings, to enhance your surrogate models.

Key insights

LLM-driven kernel discovery via a two-stage process and LOO-CRPS significantly improves high-dimensional Bayesian optimization.

Principles

• Decompose LLM tasks for functional diversity in code generation.
• Penalize overfitting with LOO-CRPS for robust kernel selection.
• Tailor initial kernel populations for high-dimensional BO inductive biases.

Method

An LLM proposes mathematical kernel forms, a second LLM converts them to executable code, then validate and select using LOO-CRPS, updating the population iteratively.

In practice

• Apply two-stage LLM generation for diverse kernel code.
• Use LOO-CRPS to select robust GP kernels.
• Explore geometric warping and non-stationary kernel compositions.

Topics

• Bayesian Optimization
• Gaussian Processes
• Kernel Design
• Large Language Models
• High-dimensional Optimization
• Evolutionary Algorithms
• Hyperparameter Optimization

Code references

• Shin-woocheol/Kernel_discovery

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Unleashing LLMs in Bayesian Optimization: Preference-Guided Framework for Scientific Discovery — LGBO integrates LLM semantic reasoning into every BO iteration via a preference mechanism, accelerating scientific discovery.
• Beyond Cross-Validation: Adaptive Parameter Selection for Kernel-Based Gradient Descents — This paper proposes a novel Hybrid Selection Strategy (HSS) for Kernel-Based Gradient Descent (KGD) algorithms, which integrates bias-variance analysis with the splitting method for adaptive parameter selection.
• Import AI 439: AI kernels; decentralized training; and universal representations — AI systems can automate and accelerate complex software optimization tasks, yielding significant performance gains.
• Mining Useful General Data for Low-Resource Domain Adaptation — Neural Tangent Kernels can effectively select high-value auxiliary data for low-resource LLM domain adaptation.
• State-of-art minibatches via novel DPP kernels: discretization, wavelets, and rough objectives — Wavelet-based DPPs and a novel discretization pipeline significantly enhance minibatch and coreset sampling efficiency for diverse ML tasks.
• From SGD to Muon: Adaptive Optimization via Schatten-p Norms — Dynamically adapting LMO geometry from runtime data improves optimizer performance and design.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.LG updates on arXiv.org.