Efficient Long-Horizon Learning for Learned Optimization

2026-07-07 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Efficient Long-hOrizon (ELO) learning is a new meta-training algorithm designed to overcome limitations in learned optimizers (LOs). Current LOs struggle with scaling meta-training to long-horizon inner problems and often fail to outperform hand-designed optimizers like Adam and Muon. ELO addresses this by reallocating redundant meta-training compute to longer failure regimes, enabling efficient long-horizon learning. It also employs decoupled progressive expert supervision, providing stable meta-learning signals that enhance LO generalization. Empirical studies evaluated ELO on both element-wise and matrix-based LOs across language modeling tasks (GPT-2-124M/350M on FineWeb) and image classification (ViT-B/16, ResNet-50 on ImageNet-1K). ELO significantly improves long-unroll performance and out-of-distribution generalization, with ELO-Celo2 consistently surpassing AdamW and remaining competitive with Muon. Notably, ELO baselines require less than 7 H100 GPU-hours for meta-training.

Key takeaway

For Machine Learning Engineers developing or deploying learned optimizers, ELO learning offers a significant advancement. You should consider integrating ELO to improve long-horizon performance and out-of-distribution generalization, especially given its demonstrated ability to outperform AdamW and competitive performance against Muon. Its low meta-training cost, under 7 H100 GPU-hours, makes it a practical choice for enhancing optimizer capabilities in language modeling and image classification tasks.

Key insights

ELO learning efficiently meta-trains learned optimizers for long-horizon tasks, outperforming hand-designed optimizers with improved generalization.

Principles

• Reallocate compute to longer failure regimes.
• Use decoupled progressive expert supervision.
• Improve generalization through stable meta-learning.

Method

ELO reallocates meta-training compute for long-horizon learning and enforces decoupled progressive expert supervision to provide stable meta-learning signals, improving generalization of learned optimizers.

In practice

• Apply ELO to element-wise LOs.
• Apply ELO to matrix-based LOs.
• Evaluate on GPT-2-124M/350M or ViT-B/16.

Topics

• Learned Optimization
• Meta-learning
• ELO Algorithm
• GPT-2
• ViT-B/16
• H100 GPU

Best for: Research Scientist, NLP Engineer, Computer Vision Engineer, AI Scientist, Machine Learning Engineer

Related on AIssential

• The First Optimizer to Challenge Adam in a Decade Just Cut Training Costs in Half — Muon optimizer offers significant compute efficiency gains, challenging Adam's long-standing dominance in deep learning.
• MetaEvo: A Meta-Optimization Framework for Experience-Driven Agent Evolution — MetaEvo enables LLM agents to continually improve reasoning by optimizing how they learn principles from experience, not just what they store.
• OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training — OptiMer decouples CPT data ratio selection from training via post-hoc Bayesian optimization of distribution vectors.
• Meta-Learning Transformers to Improve In-Context Generalization — Training transformers on diverse small datasets improves in-context generalization and modularity over large, uncurated corpora.
• Efficient Hyperparameter Optimization for LLM Reinforcement Learning — JF-HPO efficiently optimizes LLM RL hyperparameters by jointly adapting model size and training budget, using proxy models, early stopping, and checkpointing.
• Optimizers in Deep Learning: From Gradient Descent to Adam — Optimizers guide neural network training by efficiently updating parameters to minimize loss, evolving from basic GD to adaptive Adam.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.