Structured Pruning of Large Language Models via Power Transformation and Sign-Preserving Score Aggregation with Adaptive Feature Retention

2026-07-11 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, long

Summary

A new structured pruning method for large language models (LLMs) is proposed, addressing key challenges in adapting Adaptive Feature Retention (AFR) to structured pruning. Developed by researchers from Chubu University and Panasonic Holdings, this approach tackles distribution mismatch between heterogeneous pruning scores, loss of sign information, and outlier influence during score aggregation. It integrates power transformation for nonlinear distribution alignment, sign-preserving score aggregation, and percentile-based outlier removal. Experiments on Llama-3-8B, Vicuna-v1.5-13B, and LLaVA-v1.5-13B demonstrate that the method maintains accuracy comparable to unstructured pruning, achieving up to 21.27 points improvement over naive averaging. It also enables practical inference speedups of 1.56-1.57x at a 50% pruning rate, with 35.1% parameter reduction for Llama-3-8B.

Key takeaway

For MLOps engineers deploying large language models, this structured pruning technique offers a significant path to reduce computational costs and memory footprint without sacrificing accuracy. You should consider implementing this method, particularly for models like Llama-3-8B or Vicuna-v1.5-13B, to achieve practical inference speedups of over 1.5x at 50% pruning. This approach allows efficient deployment on standard hardware, avoiding specialized sparse computation libraries.

Key insights

Structured pruning for LLMs can achieve unstructured pruning accuracy with practical speedups by addressing score heterogeneity, sign loss, and outliers.

Principles

• Align heterogeneous pruning score distributions.
• Preserve sign information for neuron-level consistency.
• Mitigate outlier influence in score aggregation.

Method

The method combines power transformation (exponent b=2.7) for SNIP scores, sign-preserving aggregation (average then absolute), and percentile-based (2nd/98th) outlier removal before combining ReFer and SNIP scores.

In practice

• Prune FFN layers in LLMs like Llama-3-8B.
• Apply ReFer-only pruning for vision encoders (e.g., CLIP).
• Achieve 1.57x speedup at 50% pruning.

Topics

• Structured Pruning
• Large Language Models
• Adaptive Feature Retention
• Model Compression
• Power Transformation
• Inference Speedup

Best for: Research Scientist, AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Structured Pruning of Large Language Models via Power Transformation and Sign-Preserving Score Aggregation with Adaptive Feature Retention — An improved structured pruning method for LLMs addresses challenges in adapting unstructured pruning techniques.
• Structured Pruning of Large Language Models via Power Transformation and Sign-Preserving Score Aggregation with Adaptive Feature Retention — Improved structured pruning for LLMs maintains accuracy while enabling practical inference speedups by addressing AFR adaptation challenges.
• Two-Stage Regularization-Based Structured Pruning for LLMs — Regularization before pruning can effectively transfer information, preserving LLM performance and reducing fine-tuning needs.
• Doing More With Less: Revisiting the Effectiveness of LLM Pruning for Test-Time Scaling — Unstructured pruning can enhance LLM test-time scaling performance, challenging prior assumptions about pruning's detrimental effects.
• Joint Structural Pruning and Mixed-Precision Quantization for LLM Compression — Jointly optimizing structural pruning and mixed-precision quantization globally improves LLM compression and performance at ultra-low bitrates.
• CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield — Efficient language model training is achieved through selective supervision, depth compression, and expert fusion.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.