Efficient Long-Horizon Learning for Learned Optimization

2026-07-09 · Source: cs.LG updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

Efficient Long-hOrizon (ELO) learning is a novel meta-training algorithm designed to enhance learned optimizers (LOs) by addressing inefficiencies in scaling to long-horizon inner problems and improving performance against hand-designed optimizers. Developed by researchers at Mila and Google DeepMind, ELO integrates a failure-aware resume buffer to reallocate compute to longer failure regimes and employs decoupled progressive expert supervision for stable meta-learning signals and improved generalization. Empirical evaluations demonstrate ELO's effectiveness across diverse tasks. ELO-Celo2 consistently outperforms AdamW on language modeling (GPT-2-124M/350M on FineWeb) and image classification (ViT-B/16, ResNet-50 on ImageNet-1K), while matching Muon on language modeling. Notably, ELO baselines achieve meta-training in under 7 H100 GPU-hours.

Key takeaway

For Machine Learning Engineers optimizing large models, ELO offers a path to surpass traditional optimizers like AdamW. You should consider integrating ELO-trained LOs, especially ELO-Celo2, into your workflow for language modeling and image classification tasks. This approach provides superior long-unroll performance and out-of-distribution generalization, with meta-training requiring less than 7 H100 GPU-hours. Evaluate ELO's failure-aware buffer and progressive expert supervision to enhance your model training efficiency and stability.

Key insights

ELO efficiently meta-trains learned optimizers for long-horizon tasks using a failure-aware buffer and progressive expert supervision.

Principles

• Reallocate compute from early-stage learning to later, failure-prone regimes.
• Stabilize long-horizon meta-training with progressive expert guidance.
• Decouple update direction and magnitude for expert imitation.

Method

ELO uses a failure-aware resume buffer to restart from difficult states, recycling compute. It stabilizes training with decoupled progressive expert supervision, fusing expert and LO trajectories and balancing direction/magnitude imitation.

In practice

• Use a failure-aware resume buffer for long-horizon optimization.
• Apply decoupled expert supervision for stable meta-training.
• Select expert optimizer based on LO architecture (e.g., Adam for element-wise, Muon for matrix-based).

Topics

• Learned Optimizers
• Meta-learning
• Long-horizon Learning
• Expert Supervision
• AdamW
• GPT-2

Code references

• xiaol827/ELO

Best for: Research Scientist, AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer

Related on AIssential

• Efficient Long-Horizon Learning for Learned Optimization — ELO learning efficiently meta-trains learned optimizers for long-horizon tasks, outperforming hand-designed optimizers with improved generalization.
• ELBO - Why Maximizing One Bound Solves Two Problems at Once — Maximizing the ELBO simultaneously optimizes model fit to data and approximates the intractable true posterior distribution.
• How Does a Large Language Model Actually Learn? — LLMs acquire complex capabilities through iterative next-token prediction and error-driven weight adjustments.
• Learning in Curved Weight Space:Exponential-Linear Weight Reparameterization for Improved Optimization — ELWR reparameterizes neural network weights with exponential-linear pathways, creating curved optimization geometry for faster loss descent.
• The First Optimizer to Challenge Adam in a Decade Just Cut Training Costs in Half — Muon optimizer offers significant compute efficiency gains, challenging Adam's long-standing dominance in deep learning.
• Efficient Hyperparameter Optimization for LLM Reinforcement Learning — JF-HPO efficiently optimizes LLM RL hyperparameters by jointly adapting model size and training budget, using proxy models, early stopping, and checkpointing.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.LG updates on arXiv.org.