Structured Pruning of Large Language Models via Power Transformation and Sign-Preserving Score Aggregation with Adaptive Feature Retention

2026-07-09 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

A new structured pruning method for large language models (LLMs) has been proposed, addressing key challenges when adapting the unstructured pruning technique, Adaptive Feature Retention (AFR), to structured pruning. The method tackles three main issues: distribution mismatch among heterogeneous pruning scores, loss of sign information crucial for optimization direction, and the influence of outliers. It unifies power transformation for nonlinear distribution alignment, sign-preserving score aggregation, and percentile-based outlier removal. Experiments on Llama-3-8B, Vicuna-v1.5-13B, and LLaVA-v1.5-13B demonstrate that this approach maintains accuracy comparable to unstructured pruning while achieving practical inference speedup through structured pruning.

Key takeaway

For machine learning engineers optimizing LLM deployment, this structured pruning method offers a viable path to enhance inference speed. You can achieve practical speedups on models like Llama-3-8B, Vicuna-v1.5-13B, and LLaVA-v1.5-13B without significantly compromising accuracy. Consider integrating these techniques to deploy more efficient and performant LLM applications, especially in resource-constrained environments.

Key insights

An improved structured pruning method for LLMs addresses challenges in adapting unstructured pruning techniques.

Principles

• Align heterogeneous pruning score distributions.
• Preserve sign information for optimization consistency.
• Remove outliers using percentile-based methods.

Method

Combines power transformation, sign-preserving score aggregation, and percentile-based outlier removal for robust structured pruning of LLMs.

In practice

• Prune Llama-3-8B for faster inference.
• Apply to Vicuna-v1.5-13B and LLaVA-v1.5-13B.

Topics

• Structured Pruning
• Large Language Models
• Model Compression
• Power Transformation
• Adaptive Feature Retention
• Inference Optimization

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Structured Pruning of Large Language Models via Power Transformation and Sign-Preserving Score Aggregation with Adaptive Feature Retention — Structured pruning for LLMs can achieve unstructured pruning accuracy with practical speedups by addressing score heterogeneity, sign loss, and outliers.
• Structured Pruning of Large Language Models via Power Transformation and Sign-Preserving Score Aggregation with Adaptive Feature Retention — Improved structured pruning for LLMs maintains accuracy while enabling practical inference speedups by addressing AFR adaptation challenges.
• Two-Stage Regularization-Based Structured Pruning for LLMs — Regularization before pruning can effectively transfer information, preserving LLM performance and reducing fine-tuning needs.
• Joint Structural Pruning and Mixed-Precision Quantization for LLM Compression — Jointly optimizing structural pruning and mixed-precision quantization globally improves LLM compression and performance at ultra-low bitrates.
• Model Distillation Guide: Compressing LLMs for Edge Efficiency — Model distillation compresses large language models into smaller, efficient versions using a teacher-student framework.
• Unified Static-Dynamic Pruning for Efficient LLM Inference — SPDP unifies static and dynamic pruning with co-designed GPU kernels and data formats for efficient LLM inference.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.