QuickReduce INT3 Quantization and Benchmarking on MI355

2026-07-13 · Source: AMD ROCm Blogs · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, long

Summary

AMD's latest blog post details the implementation and benchmarking of QuickReduce INT3 quantization on the MI355 platform, building on previous work with INT4 and FP4. QuickReduce, a high-performance all-reduce library for AMD ROCm, supports inline compression, achieving up to 2.25x speedup over RCCL on MI300X configurations. The new INT3 mode represents values in 3 bits, reducing the transmitted tile from 1152 bytes (INT4) to 896 bytes, a 22% reduction. Kernel tests on a single-node MI355 server (8x AMD Instinct MI355X GPUs) show INT3 reaching approximately 5x speedup over RCCL at TP=2 for 1 GB messages, outperforming INT4 for messages above ~4 MB. End-to-end evaluations with vLLM on five LLMs (Qwen3-32B, Qwen2.5-72B-Instruct, DeepSeek-V4-Flash, MiniMax-M2.5, Qwen3-235B-A22B) at TP=2 demonstrate INT3 reducing TTFT by 3.6% and TPOT by 3.7% on average, improving throughput by 3.7% compared to INT4. Accuracy is well-preserved across MoE models and strong dense models, though Qwen3-32B showed a notable drop.

Key takeaway

For AI Engineers optimizing LLM inference on AMD Instinct MI355, consider implementing QuickReduce INT3 quantization. You can achieve up to 3.7% higher throughput and 5x speedup over RCCL at TP=2 for large messages, especially with MoE or strong baseline models. Be mindful of potential accuracy drops on models with weaker inherent capabilities, like Qwen3-32B, and restrict INT3 usage to TP=2 for best results.

Key insights

QuickReduce INT3 quantization on MI355 significantly boosts LLM inference throughput with minimal accuracy loss for robust models.

Principles

• Inline compression reduces communication volume.
• Quantization noise impacts weaker models more.
• MoE models are more robust to quantization noise.

Method

QuickReduce INT3 uses block-level FP16 scale + symmetric quantization, splitting 3-bit values into low 2 bits ("q2w") and high 1 bit ("q1w") for packing.

In practice

• Use INT3 for large communication volumes.
• Limit INT3 to TP=2 for optimal gains.
• Set 1 MB activation threshold for INT3.

Topics

• LLM Inference
• Quantization
• QuickReduce
• AMD MI355
• Tensor Parallelism
• All-reduce

Code references

• mk1-project/quickreduce
• vllm-project/vllm
• sgl-project/sglang
• triton-lang/triton
• ROCm/pytorch

Best for: MLOps Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer, AI Engineer, AI Architect

Related on AIssential

• QuickReduce FP4 Quantization and Benchmarking on MI355 — QuickReduce with FP4/INT4 quantization on MI355 significantly accelerates large-message all-reduce operations for LLM inference.
• Qwen3.5 9B MoQ: Inside a Strong 3.6-bit GGUF — Advanced quantization and MoE architectures significantly boost LLM efficiency and performance on diverse hardware.
• Scaling MiniMax-M3 Inference with Distributed Serving and Operator Co-Design on AMD Instinct MI355X GPUs — Optimizing complex LLMs like MiniMax-M3 on AMD GPUs requires co-design across quantization, attention, speculative decoding, and distributed serving.
• Qwen3.5 Quantization: Similar Accuracy, More Thinking — Best Models and Recipes — Quantization significantly reduces LLM memory footprint while often retaining substantial accuracy, but careful layer selection is crucial.
• The Strangest Bottleneck in Modern LLMs — TiDAR unifies autoregressive and diffusion models to achieve parallel LLM inference, significantly boosting speed without sacrificing accuracy.
• Inference Optimization — How to Make LLMs Faster and Cheaper in Production — Optimizing LLM inference in production requires a stack of techniques to balance latency, throughput, and cost.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AMD ROCm Blogs.