Rightsizing open models may cut your AI inference spend

2026-07-15 · Source: Constellation Research · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Intermediate, quick

Summary

Salesforce has significantly reduced its AI inference expenditures by implementing a "rightsizing" strategy, tuning specific open-source models for individual tasks rather than relying on expensive, general-purpose models. This approach emerged 18 months ago when the company's Agentforce token bill escalated with traffic, prompting a rebuild of its AI strategy. Salesforce now uses a frontier model for multi-step processing, but the core reasoning is managed by a flexible harness that allows swapping models. This enables precision models, such as HyperClassifier and TextEval, both fine-tuned from GPT-OSS-20B, to handle specific jobs like intent detection and answer evaluation. This method optimizes price performance as open-source models improve, ensuring tasks are handled by purpose-built solutions.

Key takeaway

For AI Engineers managing inference costs, consider rightsizing your model stack. Instead of relying solely on large, expensive frontier models for every task, fine-tune open-source alternatives for specific jobs. This strategy, demonstrated by Salesforce, can significantly reduce your token bills and improve price performance. Evaluate your current AI workloads to identify opportunities for specialized, cost-effective model deployment, ensuring better returns on your AI investments.

Key insights

Rightsizing open-source models for specific tasks significantly cuts AI inference costs.

Principles

• Tune open-source models for single jobs.
• Use a harness to swap models freely.
• Optimize for price performance.

Method

Identify specific jobs, then fine-tune open-source models (e.g., from GPT-OSS-20B) on existing patterns to handle each task individually, rather than using a single frontier model for all reasoning.

In practice

• Fine-tune models for intent detection.
• Develop models for answer evaluation.
• Implement a model-swapping harness.

Topics

• AI Inference Costs
• Open-source Models
• Model Rightsizing
• Fine-tuning
• GPT-OSS-20B
• Agentforce

Best for: AI Architect, Machine Learning Engineer, NLP Engineer, MLOps Engineer, AI Engineer, Director of AI/ML

Related on AIssential

• Open-Source AI Gains Ground as Rising Costs Push Shift to Smaller Models — Open-weight AI models offer substantial cost savings and near-parity performance compared to closed models, yet remain significantly underutilized.
• Hugging Face’s CEO on why companies are done renting their AI — Companies increasingly shift from proprietary AI APIs to open source models due to scaling costs.
• Soaring Costs Prompt Fresh Interest in Open Source AI. Chinese Firms Are Way Ahead. — High costs and geopolitical risks are accelerating the shift towards open source AI, particularly benefiting Chinese models and new Western entrants.
• OpenAI Discovers New Way to Cut Inference Costs in Half — OpenAI significantly cut inference costs by optimizing existing server chips.
• Stop Renting Your Cognitive Infrastructure - Thiyagarajan Maruthavanan, Kalmantic Labs — Renting AI inference is suitable for learning, but owning infrastructure is essential for earning and scaling post-product market fit.
• GV’s Dave Munichiello On Qualcomm’s Modular Purchase, The Firm’s 10x Return And The Shift In AI Software — AI software layers are gaining value, enabling efficient model execution across heterogeneous hardware amidst scarcity and the shift to disaggregated inference.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Constellation Research.