Gate-Zero Growth: A Geometric Framework for Function-Preserving Continual Learning

2026-07-16 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Gate-zero growth introduces a function-preserving (FP) operator for continual learning, enabling the addition of new residual blocks via a zero-initialised gate. This mechanism induces rank separation in the functional Jacobian, ensuring old directions remain unchanged while new-weight directions are flat at the growth point. Function drift is controlled at O("||\u03b1||^2") and Jacobian leakage at O("||\u03b1||_\u221e"), allowing a controlled departure from the FP locus. Evaluated on a 300M\u2192857M Transformer adapted from WikiText-103 to BookCorpus, gate-zero growth achieved near-zero old-domain forgetting (\u0394_A < 0.1) under both Isolation and Freeze-Nothing operating points. This performance significantly outperforms a non-FP control, which suffered an order-of-magnitude larger forgetting. The geometric analysis also extends to LoRA, ReZero, and zero-init adapter constructions, positioning gate-zero growth as a canonical instance for safe capacity activation in continual learning.

Key takeaway

For machine learning engineers developing continual learning systems, this framework offers a robust solution to catastrophic forgetting. By implementing gate-zero growth, you can expand model capacity while preserving performance on previously learned tasks, achieving near-zero old-domain forgetting. Consider integrating this function-preserving operator to safely activate new capacity in your Transformer models, especially when adapting to new datasets like BookCorpus from WikiText-103.

Key insights

Function-preserving continual learning is achieved by adding zero-initialized gated residual blocks that control functional variation.

Principles

• Gate-zero growth induces rank separation in the functional Jacobian.
• Function drift is O("||\u03b1||^2") and Jacobian leakage O("||\u03b1||_\u221e").
• A shared local geometry governs safe capacity activation in continual learning.

Method

Gate-zero growth adds new residual blocks through a zero-initialised gate, ensuring old directions are unchanged and new functional variation is the only first-order source.

In practice

• Achieves \u0394_A < 0.1 old-domain forgetting on Transformers.
• Applicable under Isolation and Freeze-Nothing operating points.
• Covers LoRA, ReZero, and zero-init adapter constructions.

Topics

• Continual Learning
• Function-Preserving Operators
• Gate-Zero Growth
• Transformer Models
• Catastrophic Forgetting
• Jacobian Analysis

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Mitigating Forgetting in Continual Learning with Selective Gradient Projection — SFAO uses similarity-gated gradient updates to balance plasticity and stability in continual learning, reducing forgetting and memory.
• Catastrophic Forgetting is Low-Rank: A Function-Space Theory for Continual Adaptation — Catastrophic forgetting is a low-rank function-space phenomenon, predictable via a cross-task kernel in the NTK regime.
• Interference and Retention in Continual Learning — Forgetting in continual learning can be directly modeled as task interference energy, enabling replay-free retention.
• Preserving Plasticity in Continual Learning via Dynamical Isometry — Dynamical isometry is a key mechanism for preserving plasticity in continual learning.
• The Proof Is in Production: How Six Prime Numbers Solved Catastrophic Forgetting — Six prime-indexed embedding rows act as a topological invariant to solve catastrophic forgetting in neural networks.
• Lifecycle-Aware Federated Continual Learning in Mobile Autonomous Systems — A dual-timescale FCL framework improves autonomous system adaptation by preventing immediate forgetting and recovering from long-term drift.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.