Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale

2026-07-01 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A systematic study explores in-context retrieval using language models (LMs) at million-token corpus scales and extreme length-generalization. Researchers introduce BlockSearch, a 0.6B LM retriever with architectural and training modifications, demonstrating length-generalization up to 10 times beyond its training regime. However, retrieval performance collapses under more extreme extrapolation due to an "attention dilution effect," where irrelevant documents dominate the softmax denominator. To counter this, the study proposes length-aware adjustments to the attention softmax and document-level sparse attention. With these modifications, the model matches dense retrieval on benchmarks like MS MARCO and NQ at the million-token scale, while outperforming the 7 times larger concurrent model MSA. It also significantly outperforms dense retrieval, achieving a 3 times higher score on tasks like LIMIT requiring different similarity notions.

Key takeaway

For machine learning engineers designing retrieval systems for large corpora, this research indicates that in-context retrieval with language models, specifically BlockSearch, is a viable alternative to classical dense retrieval. You should investigate integrating length-aware attention mechanisms and document-level sparse attention to mitigate attention dilution, enabling LMs to match or exceed dense retrieval performance on benchmarks like MS MARCO and NQ, and significantly outperform it on tasks requiring diverse similarity notions.

Key insights

In-context retrieval with LMs is promising but requires attention control for extreme context growth.

Principles

• Attention dilution hinders LM retrieval at scale.
• Length-aware attention adjustments improve large-scale retrieval.
• In-context LMs can outperform dense retrieval on diverse similarity tasks.

Method

BlockSearch employs architectural and training modifications, enhanced by length-aware attention softmax adjustments and document-level sparse attention to manage extreme context growth.

In practice

• Investigate BlockSearch for million-token scale retrieval.
• Implement length-aware attention for large contexts.
• Evaluate LM retrievers on diverse similarity tasks.

Topics

• Language Models
• In-context Retrieval
• Attention Mechanisms
• BlockSearch
• Dense Retrieval
• MS MARCO

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Dense Contexts Are Hard Contexts: Lexical Density Limits Effective Context in LLMs — Lexical density, not just length or position, significantly limits LLM effective context windows.
• A Production System for Podcast Discovery, A Fully Open-Source Frontier Search Agent, and More! — Recent research advances information retrieval through LLMs, sparse autoencoders, and multimodal embeddings.
• A Visual Guide to Attention Variants in Modern LLMs — Modern LLM architectures prioritize efficiency and long-context handling through diverse attention mechanisms and hybrid designs.
• Think in Sentences: Explicit Sentence Boundaries Enhance Language Model's Capabilities — Explicitly modeling sentence boundaries with delimiters enhances LLM reasoning by inducing structured, sentence-by-sentence processing.
• Chunking Strategies Beyond Fixed-Size — Effective RAG chunking requires diverse strategies beyond fixed-size, balancing context, precision, and cost.
• Query-focused and Memory-aware Reranker for Long Context Processing — A new reranking framework uses attention scores for listwise relevance estimation, outperforming existing methods in long context processing.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.