A Controlled Study of Attention-Only Transformers

2026-07-20 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A controlled study investigates the necessity of feed-forward networks (FFNs) in Transformer architectures, which account for two-thirds of non-embedding parameters. Researchers developed Simple Attention Networks (SANs), attention-only decoder transformers, and compared them against standard transformers. Initial findings showed that directly removing FFNs was detrimental, with standard transformers outperforming SANs by 0.47 nats at matched depth and 0.26 nats at matched FLOPs. However, reallocating the freed parameter budget into increased attention depth largely closed this performance gap. At matched parameter counts, the difference was a mere 0.006 nats (0.27 percent of loss), a difference that shrank with larger training budgets (5B, 30B, 105B tokens) and remained near 0.02 nats across a 29x size range. The residual gap was attributed to parametric recall, where SANs struggled with knowledge-dense tasks requiring information from weights. QK-normalization was identified as critical for training deep attention-only stacks.

Key takeaway

For Machine Learning Engineers optimizing large language models, this study suggests rethinking the necessity of feed-forward layers. If your goal is to reduce model parameter count without sacrificing performance, consider reallocating the budget from FFNs to increased attention depth. This approach can achieve comparable performance, especially when paired with robust QK-normalization, potentially enabling more efficient model deployment on constrained hardware. Evaluate your model's parametric recall for knowledge-intensive applications.

Key insights

Attention-only Transformers can match standard Transformer performance by reallocating feed-forward network parameters to increased attention depth.

Principles

• Feed-forward networks are not strictly necessary for Transformer performance.
• Parameter reallocation to attention depth can compensate for FFN removal.
• QK-normalization is crucial for training deep attention-only models.

Method

Pretrain attention-only decoder transformers (SANs) against standard transformers, matching parameter count, training FLOPs, and depth (2-48 layers) across 6M-87M parameters and up to 105B tokens.

In practice

• Consider attention-only architectures for parameter efficiency.
• Prioritize QK-normalization in deep attention-only models.
• Evaluate parametric recall for knowledge-intensive tasks.

Topics

• Attention-Only Transformers
• Feed-Forward Networks
• Transformer Architecture
• Parameter Efficiency
• QK-Normalization
• Parametric Recall

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Transformers Step-by-Step Explained (Attention Is All You Need) — Transformers enable parallel processing and efficient context capture in sequential data through a dynamic attention mechanism.
• FourierQK: Spectral Preprocessing of Query-Key Projections Improves Transformer Attention — FourierQK enhances Transformer attention by applying FFT-based spectral preprocessing to query-key projections, leveraging global frequency-domain mixing.
• Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspective — Softmax attention behaves linearly in the large-prompt, infinite-token limit, simplifying theoretical analysis.
• Lifelong In-Context Learning with Transformers Requires Parametric Forms of Attention — Parametric attention is crucial for lifelong in-context learning in Transformers, overcoming quadratic memory scaling.
• Inside the Transformer: Attention Mechanisms Deep Dive — Transformers achieve deep learning stability and expressive power through specific architectural choices like residual connections and layer normalization.
• Convolution for Large Language Models — Depthwise convolutions enhance LLM locality with minimal parameters, improving context sensitivity.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.