Introducing Instella-MoE: A State-of-the-Art Fully Open Mixture-of-Experts Language Model

2026-07-24 · Source: AMD ROCm Blogs · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, long

Summary

AMD introduced Instella-MoE on July 24, 2026, a fully open Mixture-of-Experts (MoE) language model featuring 16 billion total parameters and 2.8 billion active parameters. Trained from scratch on AMD Instinct™ MI300X and MI325X GPUs using the AMD ROCm™ software stack, Instella-MoE integrates Gated Multi-head Latent Attention (Gated MLA) and FarSkip-Collective connectivity. This model demonstrates competitive performance across benchmarks, outperforming many open models with comparable or larger active parameter counts. Its multi-stage training pipeline includes pre-training, mid-training, long-context extension to 64K tokens, supervised fine-tuning, direct preference optimization, and reinforcement learning. FarSkip-Collective notably accelerated pre-training by 12.7% and reduced inference Time to First Token by up to 39.2%. AMD has released all model weights, training configurations, data mixtures, and code under a Research RAIL license.

Key takeaway

For Machine Learning Engineers developing open-source LLMs, Instella-MoE offers a robust blueprint for efficient MoE model training and deployment on AMD hardware. You should consider integrating architectural innovations like Gated MLA and FarSkip-Collective to optimize training speed and inference latency. Explore its multi-stage pipeline, especially DPO and RL techniques, to refine your models' instruction following and reasoning. Utilize the full open-source release for reproducibility and further development.

Key insights

AMD's Instella-MoE demonstrates efficient, high-performance open MoE LLM training and inference on AMD hardware.

Principles

• Sparse MoE designs balance performance and compute.
• Architectural innovations enhance training and inference efficiency.
• Multi-stage training pipelines progressively refine model capabilities.

Method

Instella-MoE uses a multi-stage pipeline: pre-training, mid-training, long-context extension, SFT (with feedback-driven curation), DPO (disabling router bias/load-balancing), and RL (with MOPD).

In practice

• Utilize Gated MLA for selective attention responses.
• Implement FarSkip-Collective for communication-computation overlap.
• Disable DPO router bias for MoE training stability.

Topics

• Instella-MoE
• Mixture-of-Experts
• AMD ROCm Ecosystem
• LLM Architectural Innovations
• Multi-stage LLM Training
• Open-Source LLMs

Code references

• AMD-AGI/Instella-MoE
• AMD-AGI/FarSkip-Collective

Best for: AI Architect, AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Dots.llm1.inst: The Instruction-Tuned Mixture-of-Experts Language Model You Need to Know About — `dots.llm1.inst` is an efficient MoE LLM offering Qwen2.5-72B-level performance with 14B active parameters for multilingual instruction-following.
• Latest open artifacts (#18): Arcee's 400B MoE, LiquidAI's underrated 1B model, new Kimi, and anticipation of a busy month — Smaller open models are achieving competitive performance against larger counterparts through extensive pretraining and architectural innovations.
• The Complete Guide to How LLMs Work and What Makes Each Model Different — LLM innovation extends beyond parameter count, focusing on architectural efficiency, reasoning control, and multimodal integration for diverse applications.
• What are Mixture-of-Experts Models | ft. Aritra — MOEs enable sparse activation in dense transformer architectures, significantly boosting inference efficiency and speed.
• DeepSeek-V3: A Large-Scale MoE Pretraining Benchmark for MLPerf Training v6.0 — MLPerf Training v6.0 now benchmarks MoE LLMs, reflecting industry shifts to sparse computation and specialized architectures.
• The Architecture Behind Open-Source LLMs — MoE architectures and diverse training strategies drive rapid progress in open-weight LLMs, optimizing cost and performance.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AMD ROCm Blogs.