Scaling Adaptive Depth with Norm-Agnostic Residual Networks

2026-06-15 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new Norm-Agnostic Residual (NAG) architecture addresses the issue of residual stream norm growth in deep learning models, which typically diminishes the impact of later layer updates. NAG separates magnitude from directional information in the residual stream, ensuring meaningful contributions from all layers and preventing systematic suppression as depth increases. This architecture introduces negligible additional parameters and uses simple, kernel-fusible operations, maintaining training efficiency. NAG demonstrates superior performance over baseline Transformers, with gains becoming more substantial in deeper models. Furthermore, NAG enables an interpretable Mixture-of-Depths (MoD) mechanism that adaptively skips attention and MLP layers. This MoD can serve as a post-training accuracy-compute tradeoff or a pretraining-time scaling strategy. Experiments show that moderate MoD rates of approximately 20%-25% achieve full-depth baseline performance under equal training compute, significantly reducing executed layer parameters and forward-pass FLOPs. This establishes sparsity in depth as a novel scaling axis for developing very deep, FLOP-efficient models.

Key takeaway

For Machine Learning Engineers designing deep Transformer models, the NAG architecture and its Mixture-of-Depths (MoD) mechanism offer a critical path to overcome depth limitations. You can train significantly deeper models without performance degradation by preventing residual norm growth. Consider implementing MoD to achieve substantial FLOP reductions (20%-25%) while maintaining baseline accuracy, allowing you to reinvest compute into training on more tokens for improved efficiency and scalability.

Key insights

NAG architecture prevents residual norm growth, enabling deeper, more effective models via a Mixture-of-Depths mechanism.

Principles

• Separating magnitude from direction in residuals preserves layer impact.
• Adaptive depth skipping (MoD) offers compute-accuracy tradeoffs.
• Depth sparsity is a new axis for FLOP-efficient model scaling.

Method

NAG separates residual stream magnitude from directional information. MoD adaptively skips attention and MLP layers, reinvesting saved compute into more tokens during iso-FLOP training.

In practice

• Implement NAG for training deeper Transformers.
• Use MoD for post-training accuracy-compute tradeoffs.
• Apply MoD pretraining for fixed-compute scaling.

Topics

• Norm-Agnostic Residual Networks
• Mixture-of-Depths
• Transformer Architectures
• Deep Learning Scaling
• FLOP Efficiency
• Model Sparsity

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• Finding Sparse Subnetworks in One Training Cycle via Progressive Magnitude-Based Pruning — Progressive magnitude-based pruning enables effective neural network sparsification within a single training cycle.
• CNN Architecture Evolution: ResNet → EfficientNet → ConvNeXt — What Actually Changed? — CNN performance gains are equally driven by architectural innovation and advanced training recipes.
• Spectrum-Adaptive Generalization Bounds for Trained Deep Transformers — Spectrum-adaptive generalization bounds for Transformers improve depth and dimension scaling by allowing post hoc Schatten index selection.
• Scaling LLMs horizontally: hidden-state coupling without weight modification [R] — Residual Coupling enables horizontal scaling of frozen LLMs via linear bridges, enhancing performance without weight modification.
• Surrogate models for Rock-Fluid Interaction: A Grid-Size-Invariant Approach — Grid-size-invariant neural networks offer efficient, scalable surrogate modeling for complex rock-fluid interactions.
• How a 2021 Quantization Algorithm Quietly Outperforms Its 2026 Successor — Optimal analytical scaling in vector quantization significantly improves accuracy and efficiency over fixed scaling.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.