Accelerating LLM Inference with Prompt Caching for Open‑Source Models on Databricks

2026-05-22 · Source: Databricks · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Intermediate, quick

Summary

Databricks has extended its built-in prompt caching capability to open-source large language models (LLMs) hosted on its Foundation Model APIs, including GPT-OSS 20B and 120B, Gemma 3 12B, and Llama 3.1 8B and 3.3 70B. This feature, previously available for proprietary models like GPT, Gemini, and Claude, automatically reuses identical prompt prefixes to eliminate redundant compute cycles during LLM inference. The implementation is implicit, requiring no customer configuration, and ensures prompt caches are isolated, volatile, and never persisted for security. Initial rollout on GPT-OSS models demonstrated significant real-world gains, including a 2.5x increase in per-replica input-token throughput and a 3x reduction in P50 latency, even with a relatively low 30% cache hit ratio. This enhancement aims to lower latency, increase throughput, and reduce costs for various workloads.

Key takeaway

For MLOps Engineers deploying open-source LLMs on Databricks, this automatic prompt caching feature offers substantial performance and cost benefits without requiring any configuration. Your inference pipelines for real-time chat, batch processing, or AI agents will see improved throughput and reduced latency. Consider deploying your next OSS model with Databricks' Foundation Model APIs to leverage these built-in optimizations and observe performance metrics climb.

Key insights

Prompt caching for LLMs significantly reduces inference latency and increases throughput by reusing identical prompt prefixes.

Principles

• Reprocessing identical prompt prefixes wastes compute and inflates costs.
• Prompt caching can improve model quality in specific domains.
• Implicit caching simplifies adoption and ensures security.

Method

Databricks' system automatically reuses KV caches for identical prompts, skipping the prefill stage on cache hits to improve performance.

In practice

• Apply prompt caching to shared domain-specific system prompts.
• Use for real-time chat, batch processing, or AI agent deployments.

Topics

• LLM Inference
• Prompt Caching
• Databricks Foundation Model APIs
• Open-source LLMs
• Performance Optimization
• Cost Reduction

Best for: AI Architect, CTO, VP of Engineering/Data, AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• The Complete Guide to Inference Caching in LLMs — Inference caching optimizes LLM performance and cost by reusing computation results across three distinct layers.
• DeepSeek open sources DSpark, a new framework to speed up LLM inference by up to 85% — Speculative decoding, enhanced by confidence-scheduled verification, significantly accelerates LLM inference while preserving output.
• Why Care About Prompt Caching in LLMs? — Prompt caching reduces LLM cost and latency by reusing results from repeated input tokens.
• Optimize, deploy, and benchmark an open-source LLM with vLLM — Efficient LLM inference is achieved through specialized memory management and serving systems like vLLM.
• What is Prompt Caching? Optimize LLM Latency with AI Transformers — Prompt caching stores precomputed LLM input representations (KV pairs) to reduce latency and cost for repeated prompt prefixes.
• Large model inference container – latest capabilities and performance enhancements — LMCache and EAGLE speculative decoding significantly reduce LLM inference costs and latency by reusing KV caches and accelerating token generation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Databricks.