GaugeQuant: Online Learning of Quantization-Optimal Bases from LLM Symmetries

2026-07-25 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, long

Summary

GaugeQuant is an in-training quantization method designed to reduce performance degradation in Large Language Models by exploiting their internal continuous symmetries. It introduces a LogSumExp term into the training loss, which explicitly breaks these symmetries to learn a quantization-optimal basis that minimizes activation outliers. A stop-gradient operator ensures that only rotation matrices are updated, preserving the language modeling objective. This approach requires no calibration data or quantization simulation and adds minimal training overhead. Empirical results show significant perplexity reductions: on LLaMA-2 7B, W4A4 quantization (group size 128) improved from 8.22 to 6.73, and W4A16 improved from 11.16 to 5.45. For Qwen-2.5 0.5B, W4A4 g128 perplexity dropped from 187.8 to 61.2, and W4A16 from 25.62 to 16.76.

Key takeaway

For Machine Learning Engineers deploying Large Language Models, GaugeQuant provides a compelling in-training solution to improve quantized model performance. You can integrate this method into your training pipelines to learn quantization-optimal bases, significantly reducing perplexity degradation without needing calibration data or complex post-training simulations. This approach yields models inherently more robust to quantization, allowing you to achieve efficient inference with lower memory footprints. Consider adopting GaugeQuant to produce more deployment-ready LLMs directly from your training runs.

Key insights

In-training symmetry breaking via a LogSumExp loss term optimizes quantization bases for LLMs.

Principles

• Transformer outputs are invariant to internal continuous symmetries, but quantization is not.
• Minimizing the L_infinity norm of activations directly reduces quantization error.
• O(d) is the maximal continuous symmetry group for weight decay regularization.

Method

Integrate a LogSumExp loss term with a stop-gradient operator into the language modeling objective to update rotation matrices via the Cayley transform, using block-diagonal rotations for MLPs.

In practice

• Integrate GaugeQuant into LLM pretraining or continued training for quantization-friendly models.
• Apply the gauge framework to address other hardware constraints like structured sparsity.

Topics

• Large Language Models
• Model Quantization
• In-training Optimization
• Transformer Symmetries
• Activation Outlier Suppression
• LLaMA-2

Code references

• MPedraBento/gauge-quant

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• GaugeQuant: Online Learning of Quantization-Optimal Bases from LLM Symmetries — GaugeQuant optimizes LLM quantization during training by breaking internal symmetries to minimize activation outliers.
• ScaleSweep: Accurate NVFP4 Post-Training Quantization of LLMs via Block Scale Initialization — ScaleSweep optimizes NVFP4 quantization by sweeping block scale candidates within derived theoretical bounds to minimize reconstruction error.
• Uncovering Symmetry Transfer in Large Language Models via Layer-Peeled Optimization — LLM training implicitly transfers data symmetries into structured geometric configurations within model parameters.
• Google Introduces TurboQuant: A New Compression Algorithm that Reduces LLM Key-Value Cache Memory by 6x and Delivers Up to 8x Speedup, All with Zero Accuracy Loss — TurboQuant offers substantial LLM KV Cache compression and speedup with no accuracy loss via data-oblivious vector quantization.
• From 32 Bits to 1.58: The Illustrated Guide to LLM Quantization — LLM quantization significantly reduces memory and accelerates inference while maintaining intelligence across various bit precisions.
• FPTQuant: Function-Preserving Transforms for LLM Quantization — Function-preserving transforms can shape LLM activation distributions for efficient INT4 quantization with minimal overhead.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.