I Cut My LLM Costs by 97% — Then Found Out My Filter Was Silently Dropping 1 in 3 Real Matches

2026-07-19 · Source: AI on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, medium

Summary

An LLM-powered Telegram bot designed to match job postings to user resumes faced significant operational costs, prompting the implementation of a keyword-based filter to reduce LLM calls by 97%. Initially, this optimization appeared successful, projecting a reduction from millions of calls weekly to a 13-minute evaluation cycle. However, rigorous validation, including a 20-mock-user experiment, revealed the filter silently dropped 32.6% of real matches, achieving only 67.4% recall. Subsequent debugging addressed issues like regex failures with symbols (e.g., "c++"), incorrect SQLAlchemy `NULL` handling, brittle compound keywords, and a narrow keyword list. These fixes improved recall to 89.1% while maintaining an 85.4% LLM cost reduction, highlighting the critical importance of measuring actual performance over just cost savings.

Key takeaway

For MLOps Engineers implementing cost-saving optimizations in LLM applications, rigorously validate changes against core performance metrics like recall, not just API call reductions. Your 97% cost cut might silently drop 1 in 3 real matches, breaking the system's primary function. Prioritize comprehensive, real-world testing over initial headline numbers to ensure optimizations don't introduce critical regressions. Implement incremental checkpointing for expensive validation runs.

Key insights

Aggressive cost optimizations can silently degrade core functionality if not rigorously validated against key performance metrics.

Principles

• Validate optimizations against actual performance metrics, not just cost.
• Small sample sizes can be unrepresentative for real-world diversity.
• Debugging requires systematic identification of specific failure points.

Method

Implement a multi-stage validation process: free sanity checks, free recall checks using existing ground truth, then a full-scale experiment with LLM scoring both with and without the filter.

In practice

• Use proper regex lookaround boundaries for symbol-containing keywords.
• Ensure database `NULL` values are correctly handled, not as strings.
• Broaden keyword extraction to include role-title variants and atomic terms.

Topics

• LLM Cost Optimization
• Recall Measurement
• Keyword Filtering
• A/B Testing
• MLOps Validation
• Data Quality

Best for: AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Cost Savings in LLM Applications — Optimize LLM costs by systematically controlling execution across the application pipeline, targeting "Cost Per Successful Task."
• When LLMs get significantly worse: A statistical approach to detect model degradations — A statistical framework using McNemar's test reliably detects LLM accuracy degradation by analyzing per-sample score differences.
• Think Again or Think Longer? Selective Verification for Budget-Aware Reasoning — Selective verification optimizes reasoning by deciding when to re-evaluate, reducing cost and harmful answer changes.
• Notes from evaluating a customer support chat agent system: heuristic evaluators give false signal, retrieval bugs masquerade as LLM failures, and the cost/quality Pareto frontier is rarely where you think [D] — LLM-as-judge evaluation uncovers hidden RAG system flaws and identifies cost-effective, higher-quality models.
• Mastering Context Limits: How A Developer Dropped AI Token Usage by 88 Percent — Drastically reduce LLM token costs by pre-summarizing and filtering inputs before prompting.
• I Cut Claude Fable 5 Costs by 80% Without Losing Performance: 5 Practical Techniques — Low-configuration Claude Fable 5 can cut costs by over 80% while outperforming higher-tier models like Claude Opus 4.8.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI on Medium.