PALS: Percentile-Aware Layerwise Sparsity for LLM Pruning

2026-07-08 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

PALS (Percentile-Aware Layerwise Sparsity) is a novel one-shot pruning method for Large Language Models that overcomes the limitations of uniform sparsity approaches like Wanda and SparseGPT. It dynamically adjusts per-layer sparsity based on the 99th percentile of activation magnitudes, constrained to within ±5% of the overall target ratio. On LLaMA-2-7B at 50% sparsity, PALS achieved a WikiText-2 perplexity of 10.96, significantly outperforming uniform Wanda's 12.92 (mean over 9 runs, p < 0.001). The method's effectiveness is architecture-dependent, yielding marginal gains for LLaMA-3-8B and no improvement for Mistral-7B. The research also indicates that gradient-based allocation performs worse than random for discrete weight removal. PALS integrates into the pruning pipeline with negligible cost and requires no fine-tuning.

Key takeaway

For machine learning engineers optimizing LLM deployment, if you are pruning LLaMA-2-7B models, implement PALS to achieve better perplexity scores than uniform sparsity methods. Be aware that its benefits are architecture-dependent, so validate performance on LLaMA-3-8B or Mistral-7B. Avoid relying on gradient-based allocation for discrete weight pruning, as it performs worse than random.

Key insights

PALS adaptively prunes LLM layers based on activation percentiles, outperforming uniform sparsity for specific architectures.

Principles

• Transformer layer importance varies significantly.
• Gradient magnitude is a poor predictor for discrete weight removal impact.

Method

PALS adjusts per-layer sparsity using the 99th percentile of activation magnitudes, constrained to ±5% of the target ratio.

In practice

• Consider PALS for LLaMA-2-7B model pruning.
• Avoid gradient-based methods for discrete weight removal in pruning.

Topics

• LLM Pruning
• Sparsity
• Transformer Architectures
• LLaMA-2
• Activation Magnitudes
• Model Compression

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Doing More With Less: Revisiting the Effectiveness of LLM Pruning for Test-Time Scaling — Unstructured pruning can enhance LLM test-time scaling performance, challenging prior assumptions about pruning's detrimental effects.
• Beyond Layer Importance in Layer-wise Sparsity: An Inter-Layer Perturbation-Absorption Perspective — The network's compensatory capacity, not just local layer importance, dictates optimal layer-wise sparsity in LLMs.
• Mix-and-Match Pruning: Globally Guided Layer-Wise Sparsification of DNNs — Coordinating existing pruning signals through architecture-aware, layer-wise sparsification yields more efficient and reliable DNN compression.
• SimDiff: Depth Pruning via Similarity and Difference — SimDiff improves LLM depth pruning by combining representational similarity with two transformation difference metrics.
• HARP: Hadamard-Preconditioned Adaptive Rotation Processor for Extreme LLM Quantization — HARP adaptively rotates LLM weights for extreme low-bit quantization, improving accuracy and inference speed over fixed methods.
• From Layers to Submodules: Rethinking Granularity in Replacement-Based LLM Compression — LLM redundancy is submodule-level and non-contiguous, enabling better compression via fitted residual bypasses.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.