The Reuse Hierarchy: How Modern LLMs Get Fast and Cheap

2026-07-19 · Source: LLM on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Advanced, long

Summary

A new framework, "The Reuse Hierarchy," introduces a unified mental model for optimizing Large Language Model (LLM) systems to reduce cost and latency. This model categorizes various caching and acceleration techniques into six layers, spanning from application-level strategies like Agent Memory, RAG Cache, and Semantic Cache, to architectural and inference optimizations such as Prompt/Prefix Cache, KV Cache, and Speculative Decoding. The core principle across all techniques is avoiding redundant computation, with methods varying by the unit of work reused and the associated risk to correctness. The framework highlights that application-level techniques offer larger savings but involve a "gamble" on semantic similarity, while system-level optimizations are mathematically deterministic and introduce zero risk, providing "pure performance." This series will detail the four highest-leverage techniques: RAG caching, prompt/prefix caching, the KV cache, and speculative decoding.

Key takeaway

For AI Engineers or MLOps teams facing rising LLM costs or latency, your first move should be to identify redundant computation rather than scaling hardware. System-level optimizations like prompt/prefix caching and speculative decoding offer risk-free performance gains and should be adopted by default. For larger savings, carefully implement application-level caches, such as RAG or semantic caches, with robust guardrails to manage correctness risks.

Key insights

LLM acceleration and cost reduction stem from reusing prior computation, balancing safety and payoff.

Principles

• LLM optimization is a reuse problem, not just capacity.
• Lower reuse layers are safer, smaller wins; higher layers risk correctness for bigger gains.
• System-level optimizations are lossless; application-level ones gamble on similarity.

Method

Diagnose LLM system pain points (e.g., high TTFT, slow TPOT, OOM) to identify specific bottlenecks. Then, apply the corresponding reuse layer, considering its correctness risk.

In practice

• Use prompt/prefix caching for high TTFT on long shared prompts.
• Apply speculative decoding for slow token streams (high TPOT).
• Implement semantic response caching for repeated questions.

Topics

• LLM Optimization
• Caching Strategies
• Speculative Decoding
• RAG Caching
• KV Cache
• Inference Optimization

Best for: MLOps Engineer, AI Engineer, AI Architect

Related on AIssential

• What I Learned From Implementing LLM Architectures From Scratch (And How to Get Started) — Coding LLM architectures from scratch reveals critical design nuances and optimization strategies for memory and performance.
• Why Care About Prompt Caching in LLMs? — Prompt caching reduces LLM cost and latency by reusing results from repeated input tokens.
• Prefix Caching for Builders — Properly configured prefix caching significantly reduces LLM inference costs and latency by reusing stable prompt components.
• Can I Buy Your KV Cache? — Precomputing and reusing LLM KV caches for shared content drastically reduces redundant prefill computation and cost.
• PARTREP: Learning What to Repeat for Decoder-only LLMs — PartRep selectively repeats informative tokens to improve LLM reasoning efficiency by mitigating asymmetric information flow.
• Cost Savings in LLM Applications — Optimize LLM costs by systematically controlling execution across the application pipeline, targeting "Cost Per Successful Task."

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by LLM on Medium.