YaRN: Extending RoPE Without Breaking It

2026-03-17 · Source: NLP on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, long

Summary

YaRN (Yet another RoPE extensioN) is a highly efficient method for extending the context window of pre-trained Large Language Models, such as LLaMA 2, from 4,096 to 128,000 tokens with significantly reduced fine-tuning costs. It builds upon and unifies prior techniques like Position Interpolation and NTK-aware scaling, specifically leveraging "NTK-by-parts" for per-dimension frequency interpolation and introducing temperature scaling to mitigate attention entropy in long sequences. Experiments show YaRN achieves 99.4% Passkey retrieval accuracy at 128K context, preserves short-context performance, and offers up to 16x greater training efficiency compared to other methods. Its compatibility with Flash Attention 2 and a "Dynamic YaRN" variant for inference-time scaling make it a leading practical solution for long-context LLM applications, despite limitations regarding hyperparameter transferability and quadratic attention complexity.

Key takeaway

YaRN efficiently extends LLM context windows from 4K to 128K by integrating NTK-by-Parts interpolation and attention temperature scaling to mitigate sparsity. This approach achieves 99.4% passkey accuracy at 128K with 16x greater training efficiency and minimal short-context performance degradation. It enables practical, cost-effective adaptation of LLaMA-based models for processing extensive documents without expensive retraining.

Topics

• YaRN
• Rotary Position Embedding
• Context Window Extension
• Large Language Models
• Attention Mechanisms

Best for: AI Scientist, Research Scientist, NLP Engineer, AI Engineer, Machine Learning Engineer, AI Researcher

Related on AIssential

• EndPrompt: Efficient Long-Context Extension via Terminal Anchoring — Sparse positional supervision with a terminal anchor can efficiently extend LLM context windows without full-length training.
• Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving — SPIN unifies sparse attention and hierarchical memory to significantly boost long-context LLM serving efficiency.
• Lossless Prompt Compression via Dictionary-Encoding and In-Context Learning: Enabling Cost-Effective LLM Analysis of Repetitive Data — LLMs can learn dictionary encodings in-context, enabling lossless prompt compression without fine-tuning and preserving analytical accuracy.
• A Visual Guide to Attention Variants in Modern LLMs — Modern LLM architectures prioritize efficiency and long-context handling through diverse attention mechanisms and hybrid designs.
• Skill Weaving: Efficient LLM Improvement via Modular Skillpacks — SkillWeave uses modular, compressed "skillpacks" to efficiently specialize LLMs for multi-domain performance and faster inference.
• Inside LLMs Part 1: How Large Language Models Read, Encode, and Position Every Word You Write |… — LLMs transform text into numerical representations via tokenization, embeddings, and positional encoding.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by NLP on Medium.