FPTQuant: Function-Preserving Transforms for LLM Quantization

2026-07-09 · Source: cs.LG updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

FPTQuant, developed by Qualcomm AI Research, introduces four novel function-preserving transforms (FPTs) to address performance degradation in Large Language Models (LLMs) caused by outliers during quantization. These lightweight, expressive FPTs—including a mergeable pre-RoPE transform for queries/keys, a mergeable transform for values, a mergeable scaling transform within the MLP block, and a cheap dynamic scaling transform—maintain model function while shaping activation distributions for better quantization. FPTQuant requires no custom kernels, adds virtually no inference overhead, and enables static INT4 quantization with up to a 3.9x speed-up over FP. It demonstrates an excellent accuracy-speed trade-off, performing on par with or exceeding most prior work, and only slightly lower accuracy than a method up to 29% slower.

Key takeaway

For Machine Learning Engineers optimizing LLM inference on resource-constrained edge devices, FPTQuant offers a compelling solution for aggressive quantization. You should consider integrating FPTQuant's mergeable transforms to achieve static INT4 quantization, potentially yielding up to 3.9x speed-ups over FP with minimal accuracy loss and virtually no inference overhead. This approach prioritizes efficiency and broad hardware compatibility, making it ideal for deployments where dynamic quantization is unsupported or too costly.

Key insights

Function-preserving transforms can shape LLM activation distributions for efficient INT4 quantization with minimal overhead.

Principles

• Exploit equivariances for mergeable transforms.
• Balance transform expressivity with inference cost.
• Local optimization improves end-to-end training stability.

Method

FPTQuant uses four novel FPTs trained locally to reduce outliers and then end-to-end via student-teacher training to match full-precision model outputs, ensuring function preservation and quantization friendliness.

In practice

• Use "T_v" for value and output projection input quantization.
• Add "T_u" before Hadamard transforms for down projection input.
• Apply dynamic residual scalers for outlier-prone tokens.

Topics

• LLM Quantization
• Function-Preserving Transforms
• INT4 Inference
• Transformer Architecture
• Model Optimization
• Edge AI

Code references

• spcl/QuaRot
• Dao-AILab/fast-hadamard-transform

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Hardware Engineer

Related on AIssential

• MosaicQuant: Inlier-Outlier Disaggregation for Unified 4-Bit LLM Quantization — Unified 4-bit LLM quantization can achieve near-FP16 accuracy and speedup by disaggregating inliers and outliers.
• QuickReduce INT3 Quantization and Benchmarking on MI355 — QuickReduce INT3 quantization on MI355 significantly boosts LLM inference throughput with minimal accuracy loss for robust models.
• GaugeQuant: Online Learning of Quantization-Optimal Bases from LLM Symmetries — GaugeQuant optimizes LLM quantization during training by breaking internal symmetries to minimize activation outliers.
• PolarQuant: Leveraging Polar Transformation for Efficient Key Cache Quantization and Decoding Acceleration — PolarQuant leverages polar coordinate transformation to efficiently quantize LLM key caches, mitigating outliers and accelerating decoding via table lookups.
• How We Made a 4B Model Beat a 14B Model — Random embedding perturbations reveal hidden knowledge in small LLMs, significantly boosting accuracy and efficiency.
• True Positive Weekly #154 — The AI landscape is evolving rapidly, with innovations spanning hardware, model compression, and interpretability.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.LG updates on arXiv.org.