SLORR: Simple and Efficient In-Training Low-Rank Regularization

2026-07-09 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

SLORR is a novel, simple, stateless, and architecture-preserving framework designed for in-training low-rank regularization of neural networks. It addresses limitations of existing methods by directly regularizing original weight matrices using GPU-friendly approximations, avoiding SVDs of large matrices, architectural modifications, or stateful cached quantities. SLORR offers two main variants based on the Hoyer sparsity metric and the nuclear norm. Evaluations on ImageNet-1K, involving continued training of ResNet-50, ViT-B/16, and ViT-L/16, and pretraining of ResNet-18, demonstrated that SLORR induces compressibility with less than 8% training overhead. Furthermore, SLORR-Hoyer applied to LLM pretraining at 135M and 560M scales showed that SLORR-trained compressed models substantially preserve performance compared to unregularized models, adding less than 1% average training overhead.

Key takeaway

For Machine Learning Engineers optimizing large neural networks for deployment, SLORR offers a compelling solution to enhance model compressibility during training. If you are struggling with significant accuracy loss from post-training factorization, integrating SLORR can preserve performance substantially better than unregularized models. You should consider implementing SLORR, especially for LLM pretraining, as it adds less than 1% average training overhead while yielding more compressible and performant models.

Key insights

SLORR enables efficient in-training low-rank regularization for neural networks, improving compressibility with minimal overhead.

Principles

• Direct weight matrix regularization is effective.
• GPU-friendly approximations can replace complex operations.
• Stateless regularization simplifies integration.

Method

SLORR directly regularizes original weight matrices using GPU-friendly approximations for forward and backward passes, instantiated via Hoyer sparsity or nuclear norm.

In practice

• Apply SLORR to ResNet-50, ViT-B/16, ViT-L/16 for ImageNet-1K.
• Use SLORR-Hoyer for LLM pretraining at 135M and 560M scales.
• Integrate SLORR to reduce model size with <8% training overhead.

Topics

• Low-Rank Regularization
• Neural Network Compression
• Large Language Models
• Model Pretraining
• Training Efficiency
• ViT

Best for: AI Engineer, NLP Engineer, Computer Vision Engineer, AI Scientist, Machine Learning Engineer

Related on AIssential

• SLORR: Simple and Efficient In-Training Low-Rank Regularization — SLORR efficiently induces low-rank structure in neural networks during training without SVDs or architectural changes, improving post-compression performance.
• From Weights to Features: SAE-Guided Activation Regularization for LLM Continual Learning — SAE-guided activation regularization combats LLM catastrophic forgetting by using monosemantic features instead of polysemantic weights.
• Every Regularised Model You Have Ever Trained Is Secretly Solving a Constrained Optimisation… — Every regularized model implicitly solves a constrained optimization problem using Lagrange multipliers.
• GPT-OSS 20B: A Sparse MoE Pretraining Benchmark for MLPerf Training v6.0 — GPT-OSS 20B offers a stable, accessible benchmark for sparse Mixture-of-Experts LLM pretraining in MLPerf Training v6.0.
• Taming Momentum: Rethinking Optimizer States Through Low-Rank Approximation — LoRA-Pre reduces optimizer memory overhead by applying low-rank approximation to momentum states in large model training.
• Convex training of Lipschitz-regularized shallow neural networks — A convex restriction efficiently solves non-convex Lipschitz-regularized shallow neural network training, enhancing adversarial robustness.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.