Is Attention sink without Positional Encoding unavoidable? [D]

2026-04-30 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, medium

Summary

A developer pre-training small Transformer-based models, specifically Encoder-Decoder and Cross-attention memory-only models, consistently observes "vertical hot lines" in attention heatmaps when Positional Encoding (PE) is removed from Self-attention or Cross-attention. This pattern suggests that every query vector attends to the same key tokens, indicating a lack of dynamic, query-conditioned attention. The issue persists even with regularization attempts to spread out attention and does not resolve over tens of thousands of training steps, with loss remaining unaffected initially. The developer questions the necessity of PE in cross-attention, especially when encoder and decoder hidden states already incorporate PE from their respective self-attention mechanisms. Proposed solutions like QKNorm and SoftPick did not resolve the problem.

Key takeaway

For AI Engineers and Research Scientists developing Transformer models for text generation, if you observe "vertical hot lines" in attention heatmaps, it strongly indicates an "attention sink" problem due to missing Positional Encoding (PE). Ensure PE is consistently applied across both self-attention and cross-attention layers, as its absence can prevent dynamic token-to-token relationships from forming, leading to arbitrary attention on common words.

Key insights

Removing Positional Encoding from Transformer attention mechanisms leads to "attention sink" behavior.

Principles

• Positional Encoding is crucial for dynamic, query-conditioned attention.
• Sequence problems require positional information for statistical relationships.

In practice

• Always include Positional Encoding in self-attention for sequence tasks.
• Consider PE in cross-attention even if encoder/decoder states already have it.

Topics

• Transformer Models
• Positional Encoding
• Attention Mechanisms
• Attention Sink Problem
• Text Generation

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Transformer Architecture Explained — Transformers leverage attention mechanisms and parallel processing for efficient, context-aware sequence-to-sequence modeling.
• A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation — Attention sinks in Transformers arise from complex interactions, not single components, and can manifest through diverse circuits.
• Rethinking the Role of Positional Encoding: Sliding-Window Transformers without PE Remain Turing Complete — Sliding-window transformers can achieve Turing completeness without explicit positional encodings due to inherent window-induced asymmetry.
• Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders — Explicitly disentangling positional and semantic representations in Transformers improves positional encoding and linguistic understanding.
• The Attention Problem Nobody Has Solved — Until Now? — Subquadratic claims to solve the O(n²) attention problem, a fundamental transformer constraint since 2017.
• A new way to increase the capabilities of large language models — PaTH Attention improves LLM sequential reasoning by making positional encoding adaptive and context-aware, outperforming static methods.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.