SproutRAG: Attention-Guided Tree Search with Progressive Embeddings for Long-Document RAG

2026-06-18 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, long

Summary

SproutRAG is a novel attention-guided hierarchical Retrieval-Augmented Generation (RAG) framework designed for long documents, addressing the balance between retrieval granularity and contextual coherence. It constructs a binary chunking tree by organizing sentence-level chunks into progressively larger, semantically coherent units using learned inter-sentence attention. Unlike prior methods, SproutRAG avoids costly LLM calls during indexing or retrieval, fixed context expansion, or lossy summarization. It learns optimal attention heads and layers to capture semantic document structure, enabling multi-granularity retrieval via hierarchical beam search. Trained end-to-end with a joint objective, SproutRAG improves information efficiency (IE) by 6.1% on average across scientific, legal, and open-domain benchmarks, including 8.06 points on Dragonball and 6.83 on MS MARCO. It also demonstrates superior end-to-end performance with 4.38K online tokens per query and 193 ms latency.

Key takeaway

For Machine Learning Engineers building RAG systems for long documents, SproutRAG offers a compelling solution to enhance retrieval quality and efficiency. You should consider adopting its attention-guided hierarchical indexing and multi-granularity retrieval to overcome limitations of fixed-chunking or LLM-heavy approaches. This framework allows you to achieve superior information efficiency and end-to-end answer quality, particularly for tasks requiring cross-paragraph synthesis, while maintaining low online inference costs.

Key insights

SproutRAG uses learned attention to build a multi-granularity RAG tree, improving retrieval efficiency and coherence without LLM calls.

Principles

• Learned attention weights mitigate proximity bias in sentence transformers.
• Hierarchical beam search enables multi-granularity evidence retrieval.
• Joint training optimizes both embeddings and tree structure.

Method

Documents are split into sentence chunks, encoded by an SLLM. Learned head-layer attention weights guide bottom-up binary tree construction. Hierarchical beam search retrieves multi-granularity candidates.

In practice

• Deploy SproutRAG for RAG on long scientific or legal documents.
• Use learned attention weights to improve semantic chunking.
• Implement hierarchical beam search for multi-granularity retrieval.

Topics

• Retrieval-Augmented Generation
• Long Document Processing
• Hierarchical Retrieval
• Attention Mechanisms
• Sentence Transformers
• Information Efficiency

Code references

• AmirAbaskohi/SproutRAG
• cavedweller509/SentenceVAE

Best for: Research Scientist, AI Architect, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• SproutRAG: Attention-Guided Tree Search with Progressive Embeddings for Long-Document RAG — SproutRAG uses attention-guided tree search and progressive embeddings for multi-granularity RAG, improving information efficiency by 6.1%.
• MCompassRAG: Topic Metadata as a Semantic Compass for Paragraph-Level Retrieval — Topic metadata, selected and abstracted, can efficiently guide RAG retrieval over coarse chunks, improving precision and latency.
• A Unified Framework for Context-Aware and Relation-Aware Graph Retrieval-Augmented Generation — HyGRAG unifies context and relations in hierarchical graphs for RAG, improving multi-hop reasoning by 9.7%.
• The 89% Ceiling: Why Vector RAG is Failing and the Rise of Reasoning-Based Retrieval — Reasoning-based retrieval using hierarchical structures and LLMs significantly outperforms vector RAG by understanding document context.
• Chunking Strategies for Documents in RAG-Powered AI Summarizers — Effective document chunking is critical for RAG pipeline performance, directly impacting retrieval quality and LLM context.
• Embeddings Aren’t Magic: The Predictable Failure Modes of RAG Retrieval — Embeddings provide synonym-tolerant search but predictably fail on structural issues like OOV terms and logical operations.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.