How an Unexpected Reddit Spike Forced Me to Learn Prompt Caching the Hard Way

2026-05-22 · Source: LLM on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Intermediate, medium

Summary

ProspectAI, an open-source 6-agent investment research pipeline built on CrewAI and Anthropic Claude, experienced a 300% traffic surge in May, leading to unexpected API costs. Initially designed for learning, the project saw runs increase from 100 per month to over 200 per week, triggering frequent Anthropic billing alerts. To manage expenses without compromising quality, the developer implemented two key strategies. First, less critical agents like the Market, Technical, and Fundamental Analysts were switched to the cheaper Claude Haiku 4.5 model. Second, prompt caching was enabled by subclassing CrewAI's LLM provider to inject cache_control markers, achieving a 15.6% cache read ratio. This combined approach reduced the cost per run by approximately 60%.

Key takeaway

For AI Engineers or MLOps teams managing agentic LLM applications, you should proactively optimize API costs by combining strategic model selection with intelligent prompt caching. Instrument your multi-agent pipelines to track cache reads and writes per agent, as caching benefits vary significantly. This visibility allows you to apply caching where prefixes are stable and frequently reused, avoiding unnecessary cache write costs on agents with dynamic or short contexts, ultimately reducing your operational expenses by a substantial margin.

Key insights

Multi-agent LLM pipelines offer significant prompt caching benefits, especially when context is passed sequentially between agents.

Principles

• Cache writes cost more than standard input tokens.
• Measure caching effectiveness per agent, not globally.
• Model selection impacts cost and quality trade-offs.

Method

Subclass CrewAI's LLM provider (AnthropicCompletion) to override _format_messages_for_anthropic and inject cache_control: {"type": "ephemeral"} into system blocks or long user messages.

In practice

• Switch data processing agents to cheaper LLMs.
• Instrument pipelines to track cache reads/writes.
• Leverage sequential context passing for cache hits.

Topics

• LLM Cost Optimization
• Prompt Caching
• Multi-Agent Systems
• CrewAI
• Anthropic Claude
• MLOps

Best for: AI Engineer, MLOps Engineer, AI Architect

Related on AIssential

• Agentic AI: How to Save on Tokens — Optimizing LLM agent costs requires strategic token reuse, context minimization, intelligent model routing, and diligent context cleaning.
• Why Care About Prompt Caching in LLMs? — Prompt caching reduces LLM cost and latency by reusing results from repeated input tokens.
• Quoting Thariq Shihipar — Prompt caching is essential for cost-effective, low-latency agentic AI products.
• Prompt Caching on Claude: Cut Input Costs 78% (The Math Nobody Writes Down) — Claude's ephemeral prefix caching drastically reduces LLM input costs by reusing stable prompt segments.
• Prompt caching - Mistral AI — Prompt caching reuses shared prompt prefixes to reduce LLM inference costs and latency.
• Zero-Waste Agentic RAG: Designing Caching Architectures to Minimize Latency and LLM Costs at Scale — Two-tier caching with agentic validation significantly reduces RAG costs and latency while maintaining data freshness.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by LLM on Medium.