Semantic Caching for LLMs: FastAPI, Redis, and Embeddings

2026-04-27 · Source: PyImageSearch · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, extended

Summary

This lesson details building a semantic caching system for Large Language Model (LLM) applications using FastAPI, Redis, and embedding-based similarity search. The system employs a layered caching strategy: first, an exact-match cache for identical queries, then a semantic cache utilizing embeddings and cosine similarity for paraphrased queries, and finally, a fallback to the LLM for cache misses. The architecture, implemented with FastAPI for the API layer, Redis for storage, and Ollama for embedding generation and LLM inference, ensures that expensive LLM calls are minimized. The article explains the end-to-end request flow, from initial API validation and exact-match lookup to embedding generation, semantic search, and eventual LLM fallback, with successful LLM responses being cached for future reuse. Demonstrations confirm the system's behavior across cold requests, exact-match hits, semantic hits, and cache bypass scenarios.

Key takeaway

For AI Engineers building LLM-backed systems, implementing a layered semantic cache is crucial for optimizing performance and cost. You should prioritize exact-match caching before escalating to embedding-based semantic search, ensuring expensive LLM calls are only made when truly necessary. This approach significantly reduces latency and operational expenses, making your LLM applications more scalable and efficient.

Key insights

Semantic caching reduces LLM costs and latency by reusing responses for semantically similar queries via a layered approach.

Principles

• Prioritize cheaper operations first.
• Cache meaning, not just exact text.
• Isolate concerns for clarity and testability.

Method

Implement a layered cache: exact-match lookup, then embedding generation and semantic similarity search (cosine similarity) against cached embeddings, finally falling back to the LLM. Store LLM responses with metadata for future reuse.

In practice

• Use FastAPI for API orchestration.
• Employ Redis for cache storage.
• Utilize Ollama for local LLM and embeddings.

Topics

• Semantic Caching
• LLM Systems
• FastAPI
• Redis
• Embeddings

Best for: AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Semantic Caching for LLMs: TTLs, Confidence, and Cache Safety — Production-ready semantic caches require hardening beyond basic functionality to ensure safety and reliability.
• How to Cache vLLM Model in FastAPI for Faster Inference — Caching VLM/LLM models in FastAPI's global scope significantly reduces inference latency by avoiding repeated initialization.
• AI Broke Caching Patterns, and Most Data Systems Still Behave Like It Is 2022 — AI integration fundamentally alters caching dynamics, making "same request" ambiguous and traditional strategies ineffective.
• The Complete Guide to Inference Caching in LLMs — Inference caching optimizes LLM performance and cost by reusing computation results across three distinct layers.
• Shipping LLMs (Part 1/6): Prompt Caching vs Semantic Caching — Prompt caching is lossless and exact, while semantic caching is lossy and similarity-based, impacting correctness.
• Beyond Prompt Caching: 5 More Things You Should Cache in RAG Pipelines — Implementing diverse caching strategies across RAG pipelines significantly boosts efficiency and reduces operational costs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by PyImageSearch.