SLORR: Simple and Efficient In-Training Low-Rank Regularization

2026-07-11 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, extended

Summary

SLORR, a novel framework for in-training low-rank regularization, offers a simple, stateless, and architecture-preserving approach to enhance neural network compressibility. It avoids expensive Singular Value Decompositions (SVDs) by employing GPU-friendly approximations for Hoyer sparsity and nuclear norm regularizers. Empirical evaluations demonstrated SLORR's effectiveness across diverse models and tasks. On ImageNet-1K, it improved post-training compressibility for ResNet-50, ViT-B/16, and ViT-L/16 during continued training, and for ResNet-18 pretraining, with less than 8% training overhead. For LLM pretraining at 135M and 560M scales, SLORR-Hoyer-trained compressed models preserved performance substantially better than unregularized counterparts, adding less than 1% average training overhead.

Key takeaway

For Machine Learning Engineers optimizing model size and inference costs, SLORR provides an efficient method to enhance compressibility. You should integrate SLORR-Hoyer or SLORR-Nuc into your training pipelines. For LLMs, this adds less than 1% overhead, significantly preserving performance post-compression. This enables aggressive model size reductions without substantial accuracy degradation.

Key insights

SLORR efficiently induces low-rank structure in neural networks during training without SVDs or architectural changes, improving post-compression performance.

Principles

• Low-rank structure improves model compressibility.
• Direct spectral regularization is often too costly.
• Approximating polar factors enables efficient regularization.

Method

SLORR approximates spectral quantities for Hoyer sparsity and nuclear norm regularizers using GPU-efficient polar factor approximations (e.g., Polar Express) for forward and backward passes, applied directly to original weight matrices.

In practice

• Apply SLORR to linear and convolutional layers.
• Use 6 Polar Express iterations for efficiency.
• Consider decoupled regularization for AdamW.

Topics

• Low-Rank Regularization
• Neural Network Compression
• Large Language Models
• Vision Transformers
• Polar Express
• Training Overhead

Code references

• naiqili/EDLRR
• davidgonmar/SLORR
• pytorch/vision
• huggingface/pytorch-image-models
• XitongSystem/LoRITa

Best for: Research Scientist, AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• SLORR: Simple and Efficient In-Training Low-Rank Regularization — SLORR enables efficient in-training low-rank regularization for neural networks, improving compressibility with minimal overhead.
• A Regularization-Sharpness Tradeoff for Linear Interpolators — A regularization-sharpness tradeoff replaces bias-variance in overparameterized linear models, decomposing model selection into alignment and local perturbation effects.
• SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices — SigmaScale enhances SVD-based LLM compression by learning activation-aware scaling matrices, improving efficiency.
• Training Diagonal Linear Networks with Stochastic Sharpness-Aware Minimization — S-SAM regularizes diagonal linear networks by balancing weights, minimizing sharpness, and inducing parameter shrinkage for better generalization.
• Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy — LLM gradients exhibit a "spike-tail" anisotropy where dominant linguistic structures suppress long-tail semantic learning.
• Convex training of Lipschitz-regularized shallow neural networks — A convex restriction efficiently solves non-convex Lipschitz-regularized shallow neural network training, enhancing adversarial robustness.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.