Advanced MXFP4 Quantization: Combining Fine-Tuned Rotations with SmoothQuant for Near-Lossless Compression

2026-02-17 · Source: AMD ROCm Blogs · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, long

Summary

This article, published on February 17, 2026, details advanced MXFP4 quantization techniques for language models, specifically combining fine-tuned online rotations with SmoothQuant input channel scaling to achieve near-lossless compression. The approach addresses accuracy degradation observed in small to mid-sized models (8B to 32B parameters) when aggressively quantized to MXFP4, a 4.25-bit format supported by AMD Instinct MI350X and MI355X accelerators. The method recovers 45-55% of the accuracy drop on n-shot tasks for Qwen3-8B and Qwen3-14B models, allowing them to retain over 98% of their original BF16 accuracy. While introducing a lightweight online operation during inference, the block-diagonal structure of rotations minimizes overhead, especially for larger models. The techniques are broadly applicable to other low-precision data types like INT4 and MXFP6, with code available in AMD Quark 0.11.

Key takeaway

For NLP Engineers and AI Scientists deploying language models on AMD Instinct MI350/MI355 accelerators, adopting the combined fine-tuned online rotation and SmoothQuant scaling technique can significantly mitigate accuracy loss from MXFP4 quantization. This approach enables near-lossless compression for models like Qwen3-8B to Qwen3-32B, retaining over 98% BF16 accuracy. You should explore the AMD Quark 0.11 release for implementation and consider its integration into your inference pipelines to balance model size reduction with performance.

Key insights

Combining fine-tuned online rotations and SmoothQuant scales significantly improves MXFP4 quantization accuracy for language models.

Principles

• Outlier redistribution reduces quantization error.
• Jointly optimizing transforms and scales enhances accuracy.
• Block-diagonal rotations balance accuracy and runtime.

Method

Apply block-diagonal orthogonal transforms (rotations) to activations online, fusing inverse transforms into weights offline. Combine with SmoothQuant channel rescaling, jointly learning both rotation and scaling parameters to minimize model output error.

In practice

• Use AMD Quark 0.11 for rotation training.
• Integrate with vLLM for serving quantized models.
• Investigate fused kernels for reduced overhead.

Topics

• MXFP4 Quantization
• Online Rotations
• SmoothQuant
• LLM Inference
• AMD Instinct Accelerators

Code references

• EleutherAI/lm-evaluation-harness
• amd/Quark
• vllm-project/vllm
• ROCm/tritonBLAS
• IST-DASLab/qutlass

Best for: NLP Engineer, AI Scientist, AI Engineer, Machine Learning Engineer, Research Scientist

Related on AIssential

• Further Accelerating Kimi-K2.5 on AMD Instinct™ MI325X: W4A8 & W8A8 Quantization with AMD Quark — Quantization to W4A8 or W8A8 significantly accelerates Kimi-K2.5 inference on AMD MI325X GPUs with minimal accuracy impact.
• Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe — E2M1 FP4 training suffers "Shrinkage Bias" from geometric asymmetry, while uniform E1M2/INT4 grids offer stable, higher-quality quantization.
• ActQuant: Sub-4-bit Action-Guided Quantization for Vision-Language-Action Models — ActQuant enables sub-4-bit quantization for VLA models by action-guided mixed-precision allocation and efficient C/C++ deployment.
• HARP: Hadamard-Preconditioned Adaptive Rotation Processor for Extreme LLM Quantization — HARP adaptively rotates LLM weights for extreme low-bit quantization, improving accuracy and inference speed over fixed methods.
• Google Shrank Gemma 4 by 72% and Unsloth Fixed the 4-Bit Bug Nobody Else Caught on One 4090, and… — Google's Gemma 4 QAT and Unsloth's fix enable 4-bit quantized models to achieve near full-precision performance on consumer hardware.
• Unleashing Low-Bit Inference on Ascend NPUs: A Comprehensive Evaluation of HiFloat Formats — HiFloat formats, particularly HiF4, offer robust low-bit quantization for LLM inference on Ascend NPUs, especially at 4-bit precision.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AMD ROCm Blogs.