Loop the Loopies!

2026-07-20 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Zitian Gao and colleagues introduce Loopie, a new series of looped Transformer models, claiming it to be the most powerful looped Transformer currently available. The series includes two Mixture-of-Experts (MoE) models: a 20B-parameter model with 2B active parameters and a smaller 6B-parameter model with 0.6B active parameters. Loopie directly tackles the long-standing challenge in looped Transformers where simply increasing parameter count often yielded better results than looping for equivalent pre-training compute. Extensive ablation studies, including comparisons against a vanilla 30B-A3B model, demonstrate that Loopie significantly outperforms standard Transformer baselines when trained with the same computational budget. Furthermore, a novel post-training pipeline enhances Loopie's reasoning capabilities, enabling it to achieve gold-medal performance without external tools at the 2025 IMO and IPhO.

Key takeaway

For AI Architects and Machine Learning Engineers evaluating next-generation Transformer designs, Loopie demonstrates that looped Mixture-of-Experts architectures offer a compelling path to superior performance within constrained compute budgets. If your team is struggling with the trade-off between model size and training efficiency, consider exploring looped MoE designs and advanced post-training pipelines. This approach could yield models with significantly enhanced reasoning capabilities, as evidenced by Loopie's gold-medal performance in complex academic competitions.

Key insights

Loopie, a novel looped MoE Transformer, overcomes prior scaling limitations to achieve superior performance with equivalent compute.

Principles

• Looped Transformers can surpass vanilla models.
• MoE architectures enhance looped Transformer efficiency.
• Post-training pipelines boost reasoning abilities.

Topics

• Looped Transformers
• Mixture-of-Experts
• Transformer Architectures
• Model Scaling
• AI Reasoning
• Computational Efficiency

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• Loop the Loopies! — Looped Mixture-of-Experts Transformers can now achieve superior performance over larger vanilla models with equivalent compute.
• Mixture of Experts (MoEs) in Transformers — MoEs offer high model capacity with reduced inference costs by activating only a subset of experts per token.
• Looped Transformer !! (And More) — Looped Transformers enhance reasoning and efficiency by iteratively applying the same layer, scaling compute at test time.
• The Nine-Page Paper That Rewired Artificial Intelligence — The Transformer architecture, based on self-attention, revolutionized AI by enabling parallel processing and superior scalability.
• Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models — Expert Tying reduces MoE LLM memory footprint by almost 2x by sharing expert parameters across transformer layers with minimal performance impact.
• Why build a bigger model when you can just loop twice for twice the power? — Parallel Loop Transformers enable simultaneous model refinement, but empirical data shows two loops are often optimal, with more leading to performance regression.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.