Adaptive multi-fidelity optimization with fast learning rates

2026-04-20 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

Researchers from École Normale Supérieure, Inria Lille, and Huawei R&D, UK, introduce Kometo, a new algorithm for adaptive multi-fidelity optimization. This method addresses the challenge of optimizing locally smooth functions with a limited budget, where approximations of varying costs and biases are available. The work first establishes novel lower bounds for simple regret under different assumptions on fidelity, characterized by a cost-to-bias function. Kometo is then presented as an algorithm that achieves these optimal rates, up to logarithmic factors, without requiring prior knowledge of the function's smoothness or fidelity assumptions, improving upon previous methods like MFPDOO. Empirical results demonstrate Kometo's superior performance over existing multi-fidelity optimization techniques in several synthetic and real-world hyper-parameter tuning experiments, notably outperforming MFPDOO on three synthetic tasks and a practical text classification hyper-parameter tuning task.

Key takeaway

For machine learning engineers optimizing complex models with limited budgets, Kometo offers a robust solution for hyper-parameter tuning. Its ability to adapt to unknown function smoothness and bias characteristics, coupled with demonstrated empirical superiority over prior methods, means you can achieve better optimization outcomes without extensive problem-dependent parameter tuning. Implement Kometo to potentially reduce computational costs and improve model performance in scenarios where evaluations are expensive and gradient information is unavailable.

Key insights

Kometo optimizes functions using biased, multi-fidelity approximations without prior knowledge of smoothness or bias function.

Principles

• Lower bounds characterize problem complexity.
• Rank-based algorithms adapt to unknown bias functions.
• Adaptive methods can outperform fixed-assumption approaches.

Method

Kometo employs Zipf sampling to open a decreasing number of cells at each depth, gradually reducing fidelity. Cross-validation selects the best cell, adapting to unknown smoothness and bias parameters.

In practice

• Use Kometo for hyper-parameter tuning.
• Apply Kometo in derivative-free optimization.
• Consider Kometo for costly model evaluations.

Topics

• Multi-fidelity Optimization
• Kometo Algorithm
• Simple Regret Lower Bounds
• Derivative-Free Optimization
• Cost-to-Bias Function

Best for: NLP Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards — Augmenting PPO with multi-step likelihood ratios provides a principled way to control bias-variance in RLVR.
• Koopman Operator Identification of Model Parameter Trajectories for Temporal Domain Generalization (KOMET) — KOMET enables zero-retraining model adaptation by predicting parameter trajectories via Koopman operator identification.
• IEEE Transactions on Evolutionary Computation, Volume 30, Issue 2, April 2025 — Evolutionary algorithms and multiobjective optimization are advancing diverse fields from logistics to AI design.
• A Solver-Free Training Method for Predict-then-Optimize — A measure transformation principle enables solver-free training for predict-then-optimize models, drastically cutting training time while maintaining decision quality.
• LLMZero: Discovering Adaptive Training Strategies for RL Post-Training via LLM Agents — LLMZero uses LLM agents and tree search to adaptively optimize RL post-training, outperforming fixed schedules.
• Automated Optimization Modeling via a Localizable Error-Driven Perspective — Localized error patterns in optimization modeling enable targeted data synthesis and refined post-training for LLMs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.