Optimizing LLM Token Costs in Production: A Practical Engineering Playbook [Part 3]

2026-07-19 · Source: Machine Learning on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, medium

Summary

This article, Part 3 of a playbook, details three engineering techniques for optimizing LLM token costs in production environments, building on prior discussions of request construction. Adaptive Retrieval dynamically adjusts the amount of context sent to the LLM based on query complexity, often incorporating reranking to ensure only the most relevant documents are processed, thereby reducing unnecessary token usage. Batch Inference groups multiple non-real-time requests, such as generating embeddings for 1,000 documents into 10 batches, to minimize API call overhead and improve processing efficiency. Output Budgeting controls response length by setting "max_output_tokens" and refining prompts for conciseness, preventing the generation of unneeded information. These methods collectively enhance system design, leading to faster, cheaper, and more scalable AI applications.

Key takeaway

For MLOps Engineers optimizing LLM application costs, you should prioritize system design improvements over solely focusing on model selection. Implement adaptive retrieval to dynamically size context, ensuring only relevant information reaches the LLM. Batch non-real-time inference tasks like embedding generation to drastically reduce API overhead. Additionally, enforce output budgeting with "max_output_tokens" and precise prompts to prevent generating unnecessary, costly responses. These engineering decisions will significantly enhance your AI pipeline's efficiency, scalability, and cost-effectiveness.

Key insights

The biggest improvements in production AI often come from system design, not model selection.

Principles

• Not every query needs the same context.
• Batch non-real-time requests for efficiency.
• Concise responses improve cost and user experience.

Method

Implement adaptive retrieval by estimating query complexity to set "top_k" for document retrieval, then rerank. Use batching for offline tasks like embedding generation. Control LLM output with "max_output_tokens" and precise prompts.

In practice

• Use if/elif logic for "top_k" in RAG pipelines.
• Process document embeddings in batches of 100.
• Set "max_output_tokens" to 200 for concise answers.

Topics

• LLM Cost Optimization
• Retrieval-Augmented Generation
• Adaptive Retrieval
• Batch Inference
• Output Budgeting
• AI Pipeline Optimization

Best for: MLOps Engineer, AI Engineer, Machine Learning Engineer

Related on AIssential

• Nobody Told You AI Would Cost This Much in Production. Here’s the Real Invoice. — LLM production costs extend far beyond token pricing, driven by infrastructure, retries, and inefficient prompt management.
• Adaptive Test-Time Compute Allocation for Reasoning LLMs via Constrained Policy Optimization — Optimizing LLM test-time compute involves balancing accuracy and cost via a constrained policy.
• LLM Compression Explained: Build Faster, Efficient AI Models — AI inference, not training, drives most AI costs, making model compression essential for efficiency.
• Serving LLMs to Millions of Users The Production Playbook Part-4 — Scaling LLM systems requires architectural shifts beyond adding replicas, focusing on efficiency, cost, and resilience.
• AI for Systems: Using LLMs to Optimize Database Query Execution — LLMs can semantically optimize database query plans by correcting cardinality misestimates without engine modification.
• Optimizing LLM Token Costs in Production: A Practical Engineering Playbook [Part 1] — Inefficient LLM application architecture, not model cost, drives most production token expenses.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning on Medium.