Your AI Bill Has Two Dials. You Are Probably Only Turning One.

2026-07-13 · Source: LLM on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Intermediate, long

Summary

AI billing for metered API usage is driven by two factors: cost per token and tokens per task. While many teams prioritize reducing cost per token by selecting cheaper models, the article emphasizes that optimizing "tokens per task" offers significantly larger, multiplicative savings. This second dial is controlled by four key levers: compression, retrieval-augmented generation (RAG), caching, and smart routing. For instance, OpenAI provides automatic caching for prompts over 1,024 tokens, offering a 90% discount on cached input reads, while Anthropic offers developer-controlled caching with similar savings. Both providers also offer a 50% discount for asynchronous batch processing. The article clarifies that these optimization strategies apply specifically to pay-as-you-go API models, not flat-fee subscriptions like ChatGPT Plus, where cost optimization focuses on plan selection and seat usage.

Key takeaway

For AI Architects or MLOps Engineers managing LLM API costs, prioritize optimizing "tokens per task" over merely seeking cheaper models. You should measure input/output tokens per task to identify expensive workflows, then implement caching, compression, RAG, and smart routing. These strategies offer multiplicative savings, ensuring your unit economics improve with success rather than degrade. Focus on building token-spending discipline, as this yields greater long-term cost control than one-time model price negotiations.

Key insights

Optimizing "tokens per task" through compression, RAG, caching, and routing yields multiplicative AI cost savings.

Principles

• AI billing is a product of cost per token and tokens per task.
• Cost optimization levers stack multiplicatively, not additively.
• Match model difficulty to task, not the other way around.

Method

Implement caching by placing stable content first in prompts. Utilize batch APIs for non-real-time workloads. Apply routing, compression, and RAG within application code to reduce tokens per task.

In practice

• Use OpenAI's automatic caching for prompts >1,024 tokens.
• Implement Anthropic's `cache_control` for explicit caching.
• Move nightly jobs to batch APIs for 50% cost reduction.

Topics

• LLM Cost Optimization
• Token Management
• Prompt Caching
• Retrieval-Augmented Generation
• Model Routing
• Batch Processing

Best for: Director of AI/ML, AI Architect, MLOps Engineer

Related on AIssential

• The Token Side of FinOps: Where Your AI Bill Actually Comes From — AI inference costs are driven by operational levers like caching and batching, not just per-token price, enabling 80%+ reductions.
• Why your AI bill is bigger than it should be — LLM token costs can be drastically reduced by intelligently compressing input context before it reaches the model.
• Your AI Coding Bill Is Not a Model Problem. It’s an Orchestration Problem — The core issue in high AI coding costs is poor orchestration, not just model expense.
• The HackerNoon Newsletter: Why Cost Per Token Is the Wrong AI Metric (7/10/2026) — Total cost per successful task, not cost per token, determines AI model value.
• Optimizing LLM Token Costs in Production: A Practical Engineering Playbook [Part 1] — Inefficient LLM application architecture, not model cost, drives most production token expenses.
• Token Economics: Why AI is Getting “Cheaper” — AI cost reduction stems from optimizing token computation and making compute itself cheaper.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by LLM on Medium.