Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE

2026-07-11 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, extended

Summary

Jet-Long is a tuning-free, zero-shot method designed to efficiently extend the context window of large language models (LLMs) for applications like retrieval-augmented generation and agentic workflows. It employs a bifocal approach, combining a local RoPE-faithful window with a long-range window whose rescaling factor dynamically adapts to the current sequence length. This ensures the base model's performance is preserved at short inputs while enabling clean extrapolation at long ones, addressing position out-of-distribution issues. The method's construction, including an inclusion–exclusion attention merge and on-the-fly RoPE correction rotation, is optimized for inference. Fused into a single CuTe kernel, Jet-Long achieves up to 1.39x FlashAttention-2 throughput on H100 for long-context prefill and incurs ≤ 4% overhead for single-batch generation. It demonstrates superior performance on Qwen3-1.7B/4B/8B models up to 128K context, leading RULER by +4.79/+2.18/+2.03 pp over baselines, achieving best overall accuracy on HELMET-RAG, and lowest PG-19 perplexity. Jet-Long also generalizes to hybrid attention architectures such as Jet-Nemotron.

Key takeaway

For machine learning engineers deploying open-weight LLMs in long-context applications, Jet-Long offers a robust, tuning-free solution to extend context up to 128K tokens without sacrificing short-context performance or incurring significant inference overhead. You can adopt this method to improve RAG, coding, and agentic workflows, benefiting from its superior accuracy on benchmarks like RULER and HELMET-RAG, while maintaining near FlashAttention-2 throughput on H100 GPUs.

Key insights

Jet-Long dynamically adapts RoPE scaling for efficient, tuning-free LLM context extension, preserving short-context fidelity.

Principles

• Dynamic RoPE factor adapts to sequence length.
• Bifocal attention combines local and remote windows.

Method

Jet-Long uses a dynamic group size G for remote position mapping f(x)=⌊ x/G⌋, maintains an uncompressed KV cache with on-the-fly correction rotations, and merges three FlashAttention passes via inclusion–exclusion for prefill.

In practice

• Achieves 1.39x FA2 prefill throughput on H100.
• Generalizes to hybrid attention architectures.

Topics

• Long-Context LLMs
• Rotary Position Embedding
• Zero-Shot Context Extension
• FlashAttention
• Inference Optimization
• Qwen3

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• EndPrompt: Efficient Long-Context Extension via Terminal Anchoring — EndPrompt extends LLM context windows efficiently using short training sequences and sparse positional supervision.
• JetFlow: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Drafting — JetFlow breaks speculative decoding's scaling ceiling by combining one-forward drafting efficiency with branch-wise causal conditioning.
• Frayed RoPE and Long Inputs: A Geometric Perspective — RoPE's long-input failure stems from geometric disruption of key/query clusters, disabling attention sink tokens.
• YaRN: Extending RoPE Without Breaking It — YaRN (Yet another RoPE extensioN) is a highly efficient method for extending the context window of pre-trained Large Language Models, such as LLaMA 2, from 4,096 to 128,000 tokens with significantly r…
• Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI — Achieving 5 million token context in LLMs requires stacking multiple memory optimization techniques.
• Fast and Faithful: Real-Time Verification for Long-Document Retrieval-Augmented Generation Systems — Full-context verification for RAG systems significantly improves hallucination detection in long documents under real-time constraints.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.