Where does Absolute Position come from in decoder-only Transformers?

2026-06-06 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

RoPE-trained decoder-only Transformers exhibit absolute position leakage in their attention patterns, despite RoPE's design encoding only relative offsets. This study identifies two primary architectural sources: the causal mask's per-query softmax denominator, which depends on absolute query position, and the residual stream, where position-varying activations flow into queries and keys. The position-0 trajectory, a closed dynamical system from the initial token embedding, contributes significantly to the residual-stream component, with replacing the BOS embedding reducing this component by 40% at early queries in Llama-3.2-3B. The balance between these components varies across architectures: NTK scaling (Qwen) suppresses the residual-stream component, sliding-window attention (Mistral) allows it to accumulate with depth, and standard RoPE (Llama) sits in between. Attention sinks are token-anchored stabilizers, passing a deterministic fingerprint of the position-0 token.

Key takeaway

For AI Scientists and Machine Learning Engineers optimizing Transformer architectures, understanding absolute position leakage is crucial. You should investigate the interplay between causal masking and the residual stream, particularly the position-0 token's influence, when designing or fine-tuning models with RoPE. Consider architectural choices like NTK scaling to mitigate residual-stream leakage, especially for models where precise relative positioning is paramount, or if you need to control the stability of attention sinks.

Key insights

RoPE-trained Transformers leak absolute position via causal masks and position-0 residual stream dynamics.

Principles

• Causal mask's softmax denominator introduces absolute position dependence.
• Position-0 activation forms a fixed, deterministic trajectory.
• NTK scaling suppresses residual-stream position leakage.

In practice

• Analyze causal mask's impact on early query positions for absolute position leakage.
• Consider BOS token's role in stabilizing position-0 trajectory for model consistency.
• Evaluate NTK scaling for reducing residual-stream absolute position leakage.

Topics

• Rotary Position Embeddings
• Transformer Architectures
• Causal Masking
• Residual Stream
• Attention Sinks
• Absolute Position Leakage
• NTK Scaling

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Where does Absolute Position come from in decoder-only Transformers? — RoPE-trained Transformers gain absolute position from the causal mask and residual stream's "attention sinks."
• Rotary Positional Embeddings as Phase Modulation: Theoretical Bounds on the RoPE Base for Long-Context Transformers — RoPE behavior in long-context transformers can be analyzed as phase modulation, revealing critical theoretical bounds.
• Transformers from First Principles — Transformers use attention to process all token relationships in parallel, overcoming RNN limitations.
• Anti Mode-Collapse in Mean-Field Transformer via Auxiliary Variables — Auxiliary variables like positional encoding fundamentally prevent mode collapse in mean-field transformers.
• Rethinking the Role of Positional Encoding: Sliding-Window Transformers without PE Remain Turing Complete — Sliding-window transformers can achieve Turing completeness without explicit positional encodings due to inherent window-induced asymmetry.
• Understanding Positional Embeddings in Transformers (with Intuition and Examples) — Positional embeddings are crucial for Transformers to understand sequence order, evolving from additive to rotation-based methods.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.