Loop the Loopies!

2026-07-17 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Loopie is presented as the most powerful looped Transformer to date, comprising two Mixture-of-Experts (MoE) models: a 20B-parameter model with 2B active parameters and a 6B-parameter model with 0.6B active parameters. It successfully addresses the long-standing challenge where increasing parameter count typically outperformed looping a model N times for an N-fold compute increase. Extensive ablation studies confirm Loopie's substantial outperformance against vanilla Transformer baselines trained with identical compute budgets. Furthermore, a novel post-training pipeline endows Loopie with strong reasoning abilities, demonstrated by achieving gold-medal performance without tools at the 2025 IMO and IPhO.

Key takeaway

For Machine Learning Engineers optimizing model performance under compute constraints, you should investigate looped Mixture-of-Experts architectures like Loopie. This approach demonstrates that compute-efficient designs can outperform larger, vanilla Transformers, offering a path to deploy highly capable models with strong reasoning abilities without requiring proportional increases in parameter count. Consider exploring novel post-training pipelines to enhance such models' specific capabilities.

Key insights

Looped Mixture-of-Experts Transformers can now achieve superior performance over larger vanilla models with equivalent compute.

Principles

• Looped Transformers can surpass vanilla models for equal compute.
• Mixture-of-Experts architecture enhances looped Transformer efficiency.

In practice

• Achieve gold-medal performance in complex reasoning tasks.
• Optimize model performance under fixed compute budgets.

Topics

• Looped Transformers
• Mixture-of-Experts
• Compute Efficiency
• AI Reasoning
• Model Architecture
• Loopie

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• Loop the Loopies! — Loopie, a novel looped MoE Transformer, overcomes prior scaling limitations to achieve superior performance with equivalent compute.
• Looped Transformer !! (And More) — Looped Transformers enhance reasoning and efficiency by iteratively applying the same layer, scaling compute at test time.
• Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models — Expert Tying reduces MoE LLM memory footprint by almost 2x by sharing expert parameters across transformer layers with minimal performance impact.
• Mixture of Experts (MoEs) in Transformers — MoEs offer high model capacity with reduced inference costs by activating only a subset of experts per token.
• Why build a bigger model when you can just loop twice for twice the power? — Parallel Loop Transformers enable simultaneous model refinement, but empirical data shows two loops are often optimal, with more leading to performance regression.
• When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformers — Weight-tied looped transformers learn a linear computation frontier whose speed is priced by training budget, detectable by a novel convergence-time scaling instrument.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.