GaugeQuant: Online Learning of Quantization-Optimal Bases from LLM Symmetries

2026-07-22 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

GaugeQuant is a novel method for online learning of quantization-optimal bases in Large Language Models (LLMs) by leveraging internal continuous symmetries within Transformers. It introduces a LogSumExp term into the training loss to break these symmetries, thereby selecting a basis that effectively minimizes activation outliers. A stop-gradient operator ensures that only rotation matrices are updated, preserving the original language modeling objective. This approach requires no specific calibration data or quantization simulation and adds negligible training overhead. For the LLaMA-2 7B model, GaugeQuant significantly improves perplexity: under W4A4 quantization with group size 128, perplexity drops from 8.22 to 6.73, and under W4A16, it drops from 11.16 to 5.45.

Key takeaway

For Machine Learning Engineers deploying quantized LLMs, GaugeQuant offers a compelling in-training optimization strategy. You can achieve significant perplexity improvements, such as reducing LLaMA-2 7B W4A4 perplexity from 8.22 to 6.73, without needing calibration data or complex post-training steps. Consider integrating GaugeQuant into your LLM training pipelines to enhance model efficiency and performance with minimal overhead.

Key insights

GaugeQuant optimizes LLM quantization during training by breaking internal symmetries to minimize activation outliers.

Principles

• Transformer symmetries can be exploited for quantization.
• Online learning can select optimal quantization bases.
• Loss term can guide basis selection without altering objective.

Method

GaugeQuant adds a LogSumExp term to the loss to break Transformer symmetries, using a stop-gradient operator to update only rotation matrices, minimizing activation outliers during training.

In practice

• Apply GaugeQuant during LLM pre-training or fine-tuning.
• Achieve W4A4 quantization without calibration data.
• Improve perplexity for LLaMA-2 7B models.

Topics

• Large Language Models
• Model Quantization
• Transformer Symmetries
• Online Learning
• LLaMA-2
• Perplexity Optimization

Code references

• MPedraBento/gauge-quant

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• GaugeQuant: Online Learning of Quantization-Optimal Bases from LLM Symmetries — In-training symmetry breaking via a LogSumExp loss term optimizes quantization bases for LLMs.
• CoQuant: Joint Weight-Activation Subspace Projection for Mixed-Precision LLMs — CoQuant improves LLM quantization by jointly optimizing weight and activation subspaces for reduced output error.
• FPTQuant: Function-Preserving Transforms for LLM Quantization — Function-preserving transforms can shape LLM activation distributions for efficient INT4 quantization with minimal overhead.
• Month in 4 Papers (May 2026) — TurboQuant compresses high-dimensional vectors by simplifying quantization, preserving essential relationships for longer LLM contexts.
• How We Made a 4B Model Beat a 14B Model — Random embedding perturbations reveal hidden knowledge in small LLMs, significantly boosting accuracy and efficiency.
• The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs — Quantization significantly alters LLM internal structure and decision behavior, often undetected by standard accuracy or perplexity.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.