STARFISH: faST Accuracy Recovery in pruned networks From Internal State Healing

2026-05-31 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision & Pattern Recognition · Depth: Expert, quick

Summary

STARFISH is a novel healing method designed to efficiently recover accuracy in pruned neural networks. Pruning reduces network weights to speed up inference, often at the cost of accuracy, necessitating a subsequent healing process. STARFISH addresses this by optimizing the pruned network to align with the original network's internal state representations, utilizing a small calibration set of unlabeled examples. For networks with 50% of weights removed, STARFISH improves recovered accuracy by up to 22% compared to existing methods on ViT-based architectures. Its benefits are particularly evident under aggressive pruning; for instance, after eliminating 75% of weights in a DeiT-B network for ImageNet, STARFISH recovers 82% of the original dense accuracy using only 0.4% of the training images, significantly outperforming competing techniques that achieve only 40%.

Key takeaway

For Machine Learning Engineers optimizing model deployment, if you are struggling to maintain accuracy after network pruning, STARFISH offers a significant advancement. You should consider integrating this method into your pruning workflows, especially for ViT-based networks or aggressive pruning scenarios. STARFISH can recover substantially more accuracy with minimal unlabeled data, potentially allowing for much smaller, faster models without unacceptable performance degradation.

Key insights

STARFISH recovers pruned network accuracy by aligning its internal state representations with the original model using a small unlabeled calibration set.

Principles

• Pruning necessitates accuracy recovery.
• Internal state alignment restores accuracy.
• Unlabeled data suffices for healing.

Method

STARFISH optimizes a pruned network to match the original network's internal state representations using a tiny calibration set of unlabeled examples.

In practice

• Apply STARFISH to ViT-based networks for improved accuracy.
• Utilize STARFISH for aggressive pruning scenarios.

Topics

• Neural Network Pruning
• Model Healing
• Internal State Alignment
• ViT Networks
• DeiT-B
• Accuracy Recovery

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• this 2019 ronny chieng bit aged into a prophecy — A 2019 comedy bit predicting AI replacing human thought has become a product category, highlighting rapid technological shifts.
• The Sequence Knowledge #870: Liquid Models and the Search for a Post-Transformer Architecture — Transformers excel at cloud-scale AI but are costly for on-device applications, where liquid models offer a dynamic, efficient alternative.
• Big Tech’s Looming Capability Crisis — AI's rapid advancement can swiftly render highly skilled human capabilities obsolete.
• The art and science of hyperparameter optimization on Amazon Nova Forge — Customizing LLMs on Amazon Nova Forge requires balancing strategic choices and systematic hyperparameter tuning to prevent catastrophic forgetting.
• Expressivity of congruence-based architectures for DNNs on positive-definite matrices — Semi-orthogonality in congruence layers limits DNN expressivity for SPD matrix classification due to spectral diversity loss.
• Error Bounds for a Diffusion Model-Based Drift Estimator — Theoretical guarantees for a diffusion model-based drift estimator in SDEs are now established through an explicit risk bound.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.