CacheWeaver: Cache-Aware Evidence Ordering for Efficient Grounded RAG Inference

2026-06-18 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

CacheWeaver is a novel prompt-layer method designed to optimize Retrieval-Augmented Generation (RAG) inference by implementing cache-aware evidence ordering. RAG typically increases prompt length and prefill costs, and while serving engines like vLLM use prefix caching, this is ineffective when adjacent queries retrieve overlapping evidence in different sequences. CacheWeaver addresses this by maintaining a prefix tree of recently served evidence and employing a greedy walk to prioritize the most reusable prefix. This approach, which operates between retrieval and inference without modifying the serving engine or evidence set, significantly reduces median time-to-first-token (TTFT) by 20-33 percent across three vLLM configurations. Importantly, it achieves these gains without compromising answer quality in QA tests, with its greedy policy recovering 97.5 percent of the TTFT improvement seen with oracle ordering.

Key takeaway

For MLOps Engineers optimizing Retrieval-Augmented Generation (RAG) deployments, you should evaluate implementing cache-aware evidence ordering methods like CacheWeaver. This approach can reduce your median time-to-first-token (TTFT) by 20-33 percent, directly lowering inference costs and improving user experience, especially in high-throughput scenarios. By integrating a lightweight prompt-layer solution, you can achieve substantial efficiency gains without modifying your core serving engine or compromising answer quality.

Key insights

CacheWeaver optimizes RAG inference by reordering evidence to maximize prefix cache reuse, significantly reducing time-to-first-token.

Principles

• Evidence ordering critically impacts RAG prefill cost.
• Greedy prefix tree search yields near-optimal cache reuse.
• Prompt-layer optimization enhances serving engine efficiency.

Method

CacheWeaver constructs a prefix tree from recently served evidence sequences, then applies a greedy walk to prioritize the most reusable prefix for new RAG prompts.

In practice

• Integrate a prompt-layer reordering module.
• Optimize evidence for token overlap.
• Apply to vLLM-based RAG deployments.

Topics

• Retrieval-Augmented Generation
• LLM Inference Optimization
• Prefix Caching
• vLLM
• Time-to-First-Token
• Evidence Ordering

Best for: AI Engineer, NLP Engineer, Research Scientist, MLOps Engineer, Machine Learning Engineer, AI Scientist

Related on AIssential

• QCFuse: Query-Aware Cache Fusion via Compressed View for Efficient RAG Serving — Efficient RAG serving demands query-aware token selection that avoids pipeline stalls through compressed evidence views.
• Accelerating LLM Inference with Prompt Caching for Open‑Source Models on Databricks — Prompt caching for LLMs significantly reduces inference latency and increases throughput by reusing identical prompt prefixes.
• NeocorRAG: Less Irrelevant Information, More Explicit Evidence, and More Effective Recall via Evidence Chains — Optimizing retrieval quality through "Evidence Chains" significantly improves RAG reasoning accuracy and efficiency.
• Beyond Prompt Caching: 5 More Things You Should Cache in RAG Pipelines — Caching beyond prompts in RAG pipelines significantly reduces costs and latency by reusing query embeddings, retrieved documents, and full responses.
• Proxy-Pointer RAG: Achieving Vectorless Accuracy at Vector RAG Scale and Cost — Proxy-Pointer RAG combines structural awareness with vector search scalability for superior document retrieval.
• Answer Presence Drives RAG Rewriting Gains — RAG rewriter F1 gains are primarily driven by the gold answer string's presence, not solely by context curation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.