Why "Classic" Transformers Are Shallow and A Depth-Enabling Technique

2025-12-31 · Source: JMLR · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A 2026 paper by Yueyao Yu and Yin Zhang investigates why "classic" Transformer architectures struggle to scale to greater depths despite the Self-Attention (SA) mechanism's success since 2017. The authors identify "token similarity escalation" as the root cause, where tokens become increasingly alike after repeated SA applications. Their analysis reveals this escalation occurs at a linear rate with increasing depth, driven by invariant leading eigenspace and large spectral gaps of attention matrices. This insight leads to a proposed depth-enabling technique that surgically removes excessive token similarity without diminishing SA's overall function, unlike existing approaches. Small-scale experiments provide proof-of-concept for the technique's viability.

Key takeaway

For AI Scientists and Machine Learning Engineers designing or optimizing Transformer models, understanding token similarity escalation is crucial. This research suggests that simply stacking more layers is ineffective; instead, you should consider architectural modifications that specifically address token similarity. Implementing techniques that surgically reduce token similarity, as proposed, could enable the development of more profound and capable Transformer networks for future applications.

Key insights

Classic Transformers fail to deepen due to token similarity escalation, which a new technique can mitigate.

Principles

• Token similarity escalates linearly with Transformer depth.
• Invariant leading eigenspace drives token similarity escalation.

Method

A technique surgically removes excessive token similarity without reducing the overall role of the Self-Attention mechanism, enabling deeper Transformer models.

In practice

• Small-scale experiments validate the depth-enabling technique.
• This approach allows for trainable deeper Transformer architectures.

Topics

• Transformers
• Self-Attention
• Neural Network Depth
• Token Similarity
• Deep Learning Architectures

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• Which tokens does a hybrid model predict better? — Hybrid models predict meaning-bearing tokens better, while transformers excel at verbatim repetition and structural elements.
• AI 101: Transformers Depth Is an Addressable Dimension — New Transformer techniques enable models to dynamically search and retrieve information across layers, treating depth as an addressable memory dimension.
• Transformers from First Principles — Transformers use attention to process all token relationships in parallel, overcoming RNN limitations.
• The Nine-Page Paper That Rewired Artificial Intelligence — The Transformer architecture, based on self-attention, revolutionized AI by enabling parallel processing and superior scalability.
• To See the Unseen: on the Generalization Ability of Transformers in Symbolic Reasoning — Transformer models exhibit representational collapse of unseen token unembeddings, hindering symbolic reasoning generalization.
• Tokenization in Transformers v5: Simpler, Clearer, and More Modular — Transformers v5 tokenization decouples architecture from vocabulary, enabling greater transparency and customizability.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by JMLR.