Can Scale Save Us From Plasticity Loss in Large Language Models?

2026-06-23 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A study investigated plasticity loss, the inability of neural networks to learn new information after prior training, in modern GPT-style Transformer models. Researchers examined models ranging from 5M to 314M non-embedding parameters on a multilingual continual learning problem, using a Vietnamese probing task to measure deterioration. The findings confirm plasticity loss persists in these large language models, with its onset following a predictable sublinear scaling law. This suggests that while increasing parameter count can delay the measurable effects, it is insufficient to completely prevent plasticity loss. The phenomenon was also observed under stationary multilingual training, challenging the view that it is exclusive to abrupt task changes.

Key takeaway

For AI Scientists developing continually learning large language models, this research indicates that simply scaling up model parameters will not fundamentally solve the challenge of plasticity loss. You should explore architectural innovations or novel training paradigms that explicitly address the network's ability to retain old knowledge while efficiently integrating new information, rather than relying on brute-force scaling.

Key insights

Plasticity loss persists in LLMs, scaling sublinearly with size, suggesting scale alone won't prevent it.

Principles

• Plasticity loss affects modern Transformer LLMs.
• Its onset scales sublinearly with model size.
• Scale delays, but does not prevent, plasticity loss.

Method

Studied GPT-style Transformers on a multilingual continual learning problem, measuring deterioration on a held-out Vietnamese probing task.

In practice

• Consider plasticity loss in LLM continual learning.
• Don't rely solely on scale for adaptation.

Topics

• Plasticity Loss
• Continual Learning
• Large Language Models
• Transformers
• Scaling Laws
• Multilingual Models

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• When Data Is Scarce: Scaling Sparse Language Models with Repeated Training — Sparse training improves LLM scaling trade-offs and delays data saturation in data-scarce environments.
• Google Published a Paper That Might End the Transformer-Only LLM Era — Recurrent models can achieve growing memory by caching compressed states from sequence segments, bridging RNN and Transformer memory paradigms.
• Position: The Turing-Completeness of Real-World Autoregressive Transformers Relies Heavily on Context Management — Real-world autoregressive Transformer Turing-completeness critically depends on context management within a fixed system.
• Towards Continual Learning — Continual learning, crucial for human-like AI, involves updating models either through weight adjustments or external context/harness modifications.
• Can Large Language Models Understand Context? — LLMs struggle with nuanced context understanding, especially pre-trained dense models and those with aggressive quantization.
• A family of large language models for materials research with insights into model adaptability in continued pretraining — Domain-adapted LLMs significantly enhance materials research, but overtraining can hinder further specialization.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.