Structured Pruning of Large Language Models via Power Transformation and Sign-Preserving Score Aggregation with Adaptive Feature Retention

2026-07-09 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

A new structured pruning method for large language models (LLMs) has been developed to overcome challenges encountered when adapting Adaptive Feature Retention (AFR) from unstructured pruning. This technique specifically addresses three key issues: distribution mismatch among heterogeneous pruning scores, the loss of crucial sign information indicating optimization direction, and the impact of outlier values. The proposed unified approach integrates power transformation for nonlinear distribution alignment, sign-preserving score aggregation, and percentile-based outlier removal. Experimental evaluations on Llama-3-8B, Vicuna-v1.5-13B, and LLaVA-v1.5-13B demonstrate that this method achieves accuracy comparable to unstructured pruning while delivering practical inference speedups through structured pruning.

Key takeaway

For Machine Learning Engineers optimizing LLM deployment, this structured pruning method offers a viable path to significant inference speedups without sacrificing model accuracy. If you are struggling with adapting unstructured pruning techniques like AFR to structured settings, consider implementing power transformation, sign-preserving score aggregation, and percentile-based outlier removal. This approach allows you to achieve practical performance gains on models like Llama-3-8B, making your deployments more efficient.

Key insights

Improved structured pruning for LLMs maintains accuracy while enabling practical inference speedups by addressing AFR adaptation challenges.

Principles

• Structured pruning can match unstructured accuracy.
• Address score distribution and sign loss in pruning.
• Outlier removal improves pruning robustness.

Method

The method combines power transformation for score alignment, sign-preserving score aggregation, and percentile-based outlier removal to adapt AFR for structured LLM pruning.

In practice

• Apply power transformation for score normalization.
• Use sign-preserving aggregation for pruning scores.
• Implement percentile-based outlier filtering.

Topics

• Structured Pruning
• Large Language Models
• Model Compression
• Inference Optimization
• Power Transformation
• Llama-3-8B

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Structured Pruning of Large Language Models via Power Transformation and Sign-Preserving Score Aggregation with Adaptive Feature Retention — An improved structured pruning method for LLMs addresses challenges in adapting unstructured pruning techniques.
• Structured Pruning of Large Language Models via Power Transformation and Sign-Preserving Score Aggregation with Adaptive Feature Retention — Structured pruning for LLMs can achieve unstructured pruning accuracy with practical speedups by addressing score heterogeneity, sign loss, and outliers.
• Two-Stage Regularization-Based Structured Pruning for LLMs — Regularization before pruning can effectively transfer information, preserving LLM performance and reducing fine-tuning needs.
• Beyond FLOPs: Benchmarking Real Inference Acceleration of LLM Pruning under a GEMM-Centric Taxonomy — A GEMM-centric taxonomy unifies LLM pruning evaluation, revealing practical acceleration limits and optimal strategies across quality loss levels.
• Joint Structural Pruning and Mixed-Precision Quantization for LLM Compression — Jointly optimizing structural pruning and mixed-precision quantization globally improves LLM compression and performance at ultra-low bitrates.
• Omni-Prune: Query-Aware Unified Token Pruning for Efficient Omnimodal Large Language Models — Omni-Prune efficiently prunes audio-visual tokens in Omnimodal LLMs by integrating query-awareness and cross-modal links.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.