Token Bloat, Not Model IQ

2026-07-09 · Source: LLM on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Robotics & Autonomous Systems · Depth: Intermediate, medium

Summary

The article highlights a critical issue in AI agent development: "token bloat," where accumulated context significantly increases token usage and operational costs, even when the core task complexity remains constant. This problem is illustrated through FundlyMart, a WhatsApp B2B pharmacy bot, where a reorder conversation grew from 1,140 tokens to 4,780 tokens by turn six due to re-sent message history, raw tool outputs, and retries. The author connects this to Google's reported delay of Gemini 3.5 Pro's general availability from June to July, citing token-efficiency concerns from early testers on agentic tasks. Effective solutions are architectural, such as separating LLM reasoning context from persistent system state and stripping down raw tool outputs, rather than prompt engineering or model switching. For instance, FundlyMart reduced order-history payloads from 640 to 190 tokens.

Key takeaway

For AI Engineers building conversational or agentic AI systems, prioritize architectural solutions to manage context growth. Your focus should be on preventing token bloat by separating LLM reasoning context from persistent state and aggressively stripping unnecessary data from tool outputs. This directly impacts operational costs and agent performance, making token efficiency a critical design and procurement metric. Measure token-per-turn growth to ensure graceful degradation.

Key insights

The core problem in AI agents is token bloat from accumulated context, not model intelligence, driving up costs.

Principles

• Context growth, not reasoning growth, drives agent cost.
• Architectural fixes outperform stylistic prompt changes.
• Token efficiency is a key procurement criterion.

Method

Separate LLM reasoning context from persistent system state. Strip raw tool output to only essential data for the next reasoning step.

In practice

• Implement context discipline in agent design.
• Measure token-per-turn growth curve.
• Evaluate models on spend per completed task.

Topics

• AI Agents
• Token Efficiency
• Context Management
• Large Language Models
• Tool Calling
• Gemini 3.5 Pro

Best for: AI Engineer, Machine Learning Engineer, AI Architect

Related on AIssential

• Tokenminning: How to Get More from Your Chatbot for Less — Tokenminning systematically minimizes AI token use to reduce costs, latency, and improve output quality.
• Tokens Are the New Bytes — Tokens are becoming the architectural unit for AI, defining how language models process meaning, similar to bytes in traditional computing.
• Your AI Bill Is A Context Problem — AI budget overruns stem from "context debt" in agentic workflows, requiring a new discipline focused on context fidelity.
• How to build AI product sense — Context engineering optimizes LLM agent performance by managing context growth through offloading, reduction, isolation, and layered action spaces.
• How To Cut Your Token Budget By 80% In 3 Steps — Drastically cut AI token costs by prioritizing local inference, structured agent memory, and workflow reorchestration.
• The Context Problem — AI's "context" as a billing unit conflates capacity with coherence, leading to inflated costs and degraded model performance.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by LLM on Medium.