I Cached AI Responses to Cut My API Bill by 70%. Then My Users Started Sharing Memories.

2026-07-12 · Source: Machine Learning on Medium · Field: Technology & Digital — Software Development & Engineering, Artificial Intelligence & Machine Learning · Depth: Intermediate, quick

Summary

A solo developer significantly reduced OpenAI API costs for a Flutter voice companion app by implementing a semantic caching system. Facing monthly bills of \$847 for a project with 3,000 users, primarily due to extensive GPT-4 interactions averaging fourteen messages per session at \$0.03 per 1K tokens, the developer sought a solution. The implemented cache generates embedding vectors for user messages using "text-embedding-3-small" and queries a SQLite database for semantically similar past responses. If a similarity score above 0.92 is detected, the cached answer is returned, aiming for a 70% reduction in API expenses. While the system was built in just six hours, it began exhibiting unexpected "haunted" behavior after three weeks of operation.

Key takeaway

For AI Engineers optimizing LLM application costs, implementing semantic caching can dramatically reduce API bills, as demonstrated by a 70% cut. However, be prepared for unexpected user experience challenges, such as users "sharing memories" from cached responses, which may require careful monitoring and refinement of your caching strategy. Thoroughly test the user interaction flow to prevent unintended conversational loops or stale information.

Key insights

Semantic caching can significantly reduce LLM API costs but may introduce unforeseen user experience issues.

Principles

• LLM API costs demand optimization.
• Semantic caching uses embeddings for query similarity.
• Exact prompt matching is often insufficient.

Method

Generate message embedding with "text-embedding-3-small", search SQLite for similar queries, and return cached response if similarity exceeds 0.92.

In practice

• Implement embedding-based caching for LLM APIs.
• Utilize "text-embedding-3-small" for embeddings.
• Define a similarity threshold for cache hits.

Topics

• OpenAI API
• Semantic Caching
• LLM Cost Optimization
• Text Embeddings
• SQLite
• User Experience

Best for: AI Engineer, Software Engineer

Related on AIssential

• Shipping LLMs (Part 1/6): Prompt Caching vs Semantic Caching — Prompt caching is lossless and exact, while semantic caching is lossy and similarity-based, impacting correctness.
• Token Reduction Is Not Cost Reduction — Token reduction does not equate to cost reduction for API-based coding agents due to cache pricing and trajectory effects.
• How an Unexpected Reddit Spike Forced Me to Learn Prompt Caching the Hard Way — Multi-agent LLM pipelines offer significant prompt caching benefits, especially when context is passed sequentially between agents.
• Multi-Layer Semantic Caching for Production LLM Systems — Caching deterministic intermediate steps in LLM agentic pipelines yields greater cost and latency benefits than end-to-end caching.
• The Complete Guide to Inference Caching in LLMs — Inference caching optimizes LLM performance and cost by reusing computation results across three distinct layers.
• Prompt Caching with the OpenAI API: A Full Hands-On Python Tutorial — Prompt caching reuses repeated LLM input prefixes to enhance API app speed and cost-efficiency.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning on Medium.