Accelerating Large-Scale LLM Inference on AMD Instinct MI350X/MI355X with Eagle3 and AMD Quark

2026-07-03 · Source: AMD ROCm Blogs · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Advanced, long

Summary

AMD has significantly accelerated large language model (LLM) inference on its Instinct MI350X/MI355X GPUs by integrating Eagle3 speculative decoding with AMD Quark optimizations. Released on July 03, 2026, this initiative focuses on Kimi-K2.5 and MiniMax-M2.5 models, addressing the autoregressive decoding bottleneck. Key contributions include AMD Quark FP8 quantization for Eagle3 draft models, ROCm/vLLM backend enablement to ensure compatibility with the fast AITER MLA attention, and comprehensive InferenceX benchmark integration. Performance benchmarks for 1K/1K workloads demonstrate substantial throughput gains: Kimi-K2.5 BF16 Eagle3 achieved 1.69x to 1.90x speedup, Kimi-K2.5 FP8 Eagle3 reached 1.76x to 2.00x, and MiniMax-M2.5 BF16 Eagle3 showed 1.38x to 1.79x improvement. This full-stack enablement provides a practical path for high-throughput speculative decoding on AMD hardware.

Key takeaway

For AI engineers deploying large LLMs on AMD Instinct MI350X/MI355X GPUs, integrating Eagle3 speculative decoding offers substantial throughput improvements. You should consider adopting AMD Quark FP8 quantized draft models and ensuring your vLLM setup utilizes the AITER MLA backend for Kimi-K2.5 and MiniMax-M2.5. This approach can yield up to 2.00x speedup for 1K/1K workloads, but be aware that performance gains may degrade with longer KV caches.

Key insights

Speculative decoding with Eagle3 and AMD Quark significantly boosts LLM inference throughput on AMD Instinct MI350X/MI355X GPUs.

Principles

• Speculative decoding preserves target model output quality.
• Draft model quality directly impacts token acceptance rates.
• Quantization reduces draft model overhead for efficiency.

Method

Speculative decoding uses a smaller draft model to propose multiple tokens, which the target model then verifies in a single pass, accepting matching tokens to reduce expensive decode iterations.

In practice

• Apply FP8 quantization to speculative decoding draft models.
• Integrate vLLM with the AITER MLA backend for ROCm compatibility.
• Benchmark LLM inference using chat-template handling for realism.

Topics

• LLM Inference Acceleration
• Speculative Decoding
• AMD Instinct MI355X
• Eagle3
• AMD Quark
• FP8 Quantization
• vLLM

Code references

• vllm-project/vllm
• SemiAnalysisAI/InferenceX
• SafeAILab/EAGLE

Best for: AI Architect, MLOps Engineer, CTO, Machine Learning Engineer, AI Engineer, AI Hardware Engineer

Related on AIssential

• AMD Instinct™ GPUs MLPerf Inference v6.0 Submission — AMD's MI355X GPU and ROCm stack deliver competitive MLPerf Inference v6.0 results, especially for LLMs and multi-node setups.
• Enabling Speculative Speculative Decoding on MI300X — Speculative Speculative Decoding (SSD) hides draft latency by asynchronously pre-computing future token speculations on separate hardware.
• TileFuse: A Fused Mixed-Precision Kernel Library for Efficient Quantized LLM Inference on AMD NPUs — TileFuse enables efficient AWQ-style quantized LLM inference on AMD XDNA2 NPUs through fused mixed-precision kernels.
• Supercharging LLM inference on Google TPUs: Achieving 3X speedups with diffusion-style speculative decoding — Block diffusion speculative decoding offers substantial LLM inference speedups by generating token blocks in a single pass.
• Running 26B and 35B LLMs at Full Speed on €990 of Used Hardware — No Cloud Required — Cost-effective local LLM inference is achievable with dual secondhand GPUs, outperforming single high-end cards and revealing speculative decoding's limitations on PCIe.
• Nvidia Already Won Training. The Real Fight Is Inference — AI inference hardware is shifting from general-purpose GPUs to specialized architectures optimizing for latency and memory access.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AMD ROCm Blogs.