KronQ: LLM Quantization via Kronecker-Factored Hessian

2026-07-08 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

KronQ, a novel post-training quantization (PTQ) framework, enhances Large Language Model (LLM) compression by integrating gradient covariance into the quantization pipeline. Unlike existing second-order PTQ methods such as GPTQ, which rely solely on input activation statistics, KronQ utilizes a Kronecker-factored Hessian approximation, making the quantization loss dependent on both activation and gradient covariances. This framework introduces bidirectional incoherence processing, extending random rotation to the output dimension using gradient covariance to reduce weight magnitude variance. Additionally, KronQ derives a new sensitivity metric for inter-layer mixed-precision allocation, based on gradient and activation Hessian traces. Notably, for 2-bit weight-only quantization on LLaMA-3-70B, KronQ achieves a perplexity of 7.93 on WikiText-2, significantly outperforming GPTQ and GPTAQ, which diverge or yield degenerate quantizations with over 2000 perplexity. The framework was published on 2026-07-08.

Key takeaway

For Machine Learning Engineers optimizing LLM deployment with aggressive quantization, KronQ offers a robust solution. If you are struggling with existing PTQ methods like GPTQ failing at 2-bit weight-only quantization, KronQ's integration of gradient covariance provides significantly better perplexity. Consider evaluating KronQ for your LLaMA-3-70B models to achieve efficient 2-bit compression without the severe performance degradation seen with other techniques. This approach could enable deploying larger models on resource-constrained hardware.

Key insights

KronQ improves LLM post-training quantization by incorporating gradient covariance via Kronecker-factored Hessian approximation.

Principles

• Gradient covariance improves LLM quantization.
• Bidirectional incoherence reduces weight variance.
• Hessian traces guide mixed-precision allocation.

Method

KronQ's method involves using Kronecker-factored Hessian approximation to integrate gradient covariance, applying bidirectional incoherence processing, and deriving a sensitivity metric from Hessian traces for mixed-precision.

In practice

• Achieve 2-bit weight-only quantization on LLaMA-3-70B.
• Improve perplexity over GPTQ/GPTAQ for extreme quantization.

Topics

• LLM Quantization
• Post-Training Quantization
• Kronecker-Factored Hessian
• Gradient Covariance
• LLaMA-3-70B
• Mixed-Precision Allocation

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference — SharQ combines activation sparsity and FP4 quantization for LLM inference via an online sparse-dense decomposition.
• KroQuant: Kronecker-Structured Block Transforms for Efficient Post-Training Quantization of Diffusion Transformers — KroQuant uses Kronecker-structured block transforms and LoRaQ calibration to enable efficient W4A4 post-training quantization for Diffusion Transformers.
• PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference — PolyQ enables practical, efficient fractional-bit LLM inference on diverse edge CPUs through compiler/quantization co-design.
• C-PTQ: Fisher-weighted Channel-wise Sensitivity for Post-training Quantization of MLLMs — C-PTQ uses Fisher-weighted channel-wise sensitivity to harmonize task-specific loss and quantization error for MLLM post-training quantization.
• CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs — CAT-Q enables accurate, cost-efficient ternary quantization for LLMs via post-training methods, significantly reducing training data and time.
• A KL Lens on Quantization: Fast, Forward-Only Sensitivity for Mixed-Precision SSM-Transformer Models — KL divergence is a superior metric for identifying quantization sensitivity in hybrid SSM-Transformer LLMs.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.