How to Use Semantic Caching with Qdrant to Optimize Token Costs in Customer Support

2026-07-07 · Source: AI Advances - Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, extended

Summary

Semantic caching with Qdrant offers a solution to optimize token costs and latency in LLM-powered customer support systems by preventing redundant LLM calls for semantically similar queries. Traditional caching fails because natural language variations break exact-match systems. This approach converts queries into 384-dimensional vectors using models like BAAI/bge-small-en-v1.5, then uses Qdrant for efficient nearest-neighbor search. Benchmarks showed a 57.1% cache hit rate, reducing average response time from 2,575 ms to 15 ms (a 171x improvement) and cutting token consumption by 55.7% (1,975 tokens avoided). The system supports flexible cache invalidation via time-to-live or category-based filtering and can be extended with multi-vector retrieval for complex queries, though single-vector proved more efficient for short support questions.

Key takeaway

For MLOps Engineers deploying LLM-powered customer support, implementing semantic caching with Qdrant can significantly reduce operational costs and improve user experience. You should integrate a cache lookup early in your request pipeline, aiming for a 65-75% hit rate to achieve substantial token savings and sub-30ms response times for cached queries. Calibrate your similarity threshold using real data to balance hit rate and accuracy, and establish robust cache invalidation strategies.

Key insights

Semantic caching with Qdrant reuses LLM responses for semantically similar queries, drastically cutting costs and latency.

Principles

• Calibrate similarity thresholds with real data, not intuition.
• Simpler architectures often outperform complex ones for focused tasks.
• Cache invalidation must align with business data change frequency.

Method

Embed user queries locally, search Qdrant for similar cached vectors above a threshold. On a miss, call the LLM, then store the new query's embedding and response in Qdrant.

In practice

• Use BAAI/bge-small-en-v1.5 for local 384-dim embeddings.
• Implement TTL and category-based invalidation for cache hygiene.
• Monitor cache hit rate and embedding drift in production.

Topics

• Semantic Caching
• Qdrant
• LLM Cost Optimization
• Vector Databases
• Customer Support AI
• Embedding Models
• Cache Invalidation

Code references

• itinasharma/semantic-cache-qdrant

Best for: AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Semantic Caching for LLMs: FastAPI, Redis, and Embeddings — Semantic caching reduces LLM costs and latency by reusing responses for semantically similar queries via a layered approach.
• Nobody Told You AI Would Cost This Much in Production. Here’s the Real Invoice. — LLM production costs extend far beyond token pricing, driven by infrastructure, retries, and inefficient prompt management.
• Semantic Caching for LLMs: TTLs, Confidence, and Cache Safety — Production-ready semantic caches require hardening beyond basic functionality to ensure safety and reliability.
• How I Engineered a 1.7M Row Product Safety Database to Serve Pages in 66ms on a Single Free VM — Optimizing large-scale database performance and cost on minimal infrastructure requires strategic data management and query pattern alignment.
• AI Broke Caching Patterns, and Most Data Systems Still Behave Like It Is 2022 — AI integration fundamentally alters caching dynamics, making "same request" ambiguous and traditional strategies ineffective.
• KV Cache - Explained — Caching Key and Value vectors eliminates quadratic attention recomputation, making LLM inference linear per step.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI Advances - Medium.