Why companies are penny-pinching on tokens

2026-06-03 · Source: Semafor · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Fundamental Awareness, extended

Summary

The AI industry is experiencing a shift where companies are becoming more cautious about token spending, despite frontier models like Anthropic and OpenAI demonstrating superior performance with increased token usage. While these models offer a 10% to 20% performance edge, their high costs are prompting companies, including Microsoft, to question the ROI, especially for tasks like code generation without clear purpose. This "token stinginess" is also driven by difficulties in measuring AI's return on investment, with IBM noting "massive over-investment" and some JPMorgan employees spending more on tokens than their salaries. Experts anticipate a plateau in AI capabilities and a subsequent rapid decline in costs, a phase major tech companies like Microsoft and Google are awaiting. Meanwhile, public opposition to data center construction is growing, and energy demands for AI infrastructure are pushing for aggressive solutions.

Key takeaway

For Directors of AI/ML and VPs of Engineering evaluating AI investments, recognize that while frontier models offer performance advantages, their escalating token costs and unclear ROI demand strategic scrutiny. Focus on applications where value is directly measurable, such as sales or customer experience, and actively monitor token consumption to prevent budget overruns. Consider hybrid local/cloud AI solutions to address growing data privacy concerns and prepare for a future where AI capabilities plateau and costs rapidly optimize, favoring companies positioned for efficiency.

Key insights

Frontier AI models offer performance gains but incur significant token costs, challenging ROI measurement for enterprises.

Principles

• Performance scales with token usage in frontier models.
• AI ROI is difficult to quantify, especially for engineering.
• Public sentiment impacts AI infrastructure development.

In practice

• Prioritize AI applications with measurable outcomes like sales.
• Explore local AI processing for data privacy concerns.
• Monitor AI token usage to manage escalating costs.

Topics

• AI Economics
• Token Costs
• Frontier Models
• AI ROI
• Data Centers
• AI Governance

Best for: CTO, AI Product Manager, Product Manager, Director of AI/ML, VP of Engineering/Data, Executive

Related on AIssential

• What’s behind Anthropic’s $65B raise? — The AI sector faces a complex dynamic of hyper-growth, escalating costs, infrastructure strain, and critical safety concerns, alongside geopolitical competition.
• Let's spend 250K$ on tokens just for sake of spending — The provided content critically analyzes a "flawed logic," attributed to Jensen, which suggests that engineering value is now measured by high AI token consumption rather than efficient, optimized solutions.
• The token bill comes due: Inside the industry scramble to manage AI’s runaway costs — AI's escalating token costs necessitate new financial management standards and tools to ensure ROI amidst surging consumption.
• 🔮 X-raying OpenAI’s unit economics — Frontier AI models like GPT-5 face significant profitability challenges due to high R&D costs and short model lifespans.
• AI customer service is not ready for prime time — AI's rapid advancement brings high costs, ROI measurement difficulties, and significant safety/societal challenges, necessitating careful deployment and regulation.
• Intelligence Per Dollar — AI model evaluation is shifting from raw performance to cost-efficiency, prioritizing "intelligence per dollar."

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Semafor.