Joint Structural Pruning and Mixed-Precision Quantization for LLM Compression

2026-06-05 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new end-to-end framework addresses Large Language Model (LLM) deployment efficiency by jointly optimizing structural pruning and mixed-precision quantization. Traditional methods often optimize quantization errors per-layer, overlooking global error propagation, and apply pruning and quantization sequentially. This novel approach introduces a mixed-precision post-training quantization (PTQ) strategy that directly minimizes global error propagation across the entire model. Building on this, it develops a joint optimization method that simultaneously learns structural pruning decisions and mixed-precision quantization policies within a unified search space. Experiments show that at ultra-low precisions (1-3 bits), this quantization method reduces WikiText perplexity by up to 21% compared to leading weight-activation baselines. It also achieves up to 59% and 85% lower perplexity on WikiText and C4, respectively, against leading weight-only quantization methods, and delivers superior perplexity and reasoning performance compared to leading joint pruning-and-quantization techniques.

Key takeaway

For MLOps Engineers deploying Large Language Models with strict memory and inference latency requirements, you should prioritize compression techniques that jointly optimize pruning and quantization. This approach, which minimizes global error propagation, significantly outperforms sequential or layer-wise methods, enabling robust ultra-low precision (1-3 bits) deployments. Consider integrating unified compression frameworks to achieve superior perplexity and reasoning performance, directly impacting your model's efficiency and practical viability.

Key insights

Jointly optimizing structural pruning and mixed-precision quantization globally improves LLM compression and performance at ultra-low bitrates.

Principles

• Minimize global error propagation, not just layer-wise.
• Unify pruning and quantization into a single search space.
• Sequential optimization of compression techniques is suboptimal.

Method

The framework employs a mixed-precision PTQ strategy minimizing global error propagation, then jointly optimizes structural pruning and quantization policies within a unified search space.

In practice

• Achieve 1-3 bit LLM compression with minimal perplexity loss.
• Improve WikiText perplexity by up to 21% over leading baselines.
• Reduce C4 perplexity by up to 85% compared to weight-only methods.

Topics

• LLM Compression
• Structural Pruning
• Mixed-Precision Quantization
• Post-Training Quantization
• Global Error Minimization
• Model Optimization

Best for: Research Scientist, AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Two-Stage Regularization-Based Structured Pruning for LLMs — Regularization before pruning can effectively transfer information, preserving LLM performance and reducing fine-tuning needs.
• Does Compression Preserve Uncertainty? A Unified Benchmark for Quantized and Sparse LLMs via Conformal Prediction — LLM compression often decouples accuracy from uncertainty, necessitating uncertainty-aware benchmarking for deployment.
• Channel-Wise Mixed-Precision Quantization for Large Language Models — CMPQ adaptively quantizes LLMs channel-wise using mixed-precision and outlier protection for efficient edge deployment.
• From Layers to Submodules: Rethinking Granularity in Replacement-Based LLM Compression — LLM redundancy is submodule-level and non-contiguous, enabling better compression via fitted residual bypasses.
• XFP: Quality-Targeted Adaptive Codebook Quantization with Sparse Outlier Separation for LLM Inference — XFP is a dynamic LLM quantizer that automates codebook size and outlier budget based on operator-defined quality floors.
• HARP: Hadamard-Preconditioned Adaptive Rotation Processor for Extreme LLM Quantization — HARP adaptively rotates LLM weights for extreme low-bit quantization, improving accuracy and inference speed over fixed methods.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.