Half my eval regressions never touched the prompt file

2026-07-27 · Source: LLM on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, medium

Summary

Over the last two quarters, more than half of output regressions in a RAG service stemmed from changes that never touched the prompt directory, yet eval gates remained green. This issue arises because standard eval triggers often only monitor prompt or model changes, overlooking critical non-prompt inputs. These include tokenizer updates (e.g., Anthropic's recent models producing ~30% more tokens for the same text), retrieval index modifications, tool schema alterations, config default changes from SDK bumps, and provider model aliases resolving to newer builds. Each of these can silently degrade model output without tripping typical "paths: ["prompts/**"]" filters. The article highlights two failure modes: gates not running due to path filters, or running but passing because datasets don't exercise the affected components. It advocates for a broader approach to gate coverage.

Key takeaway

For MLOps Engineers responsible for LLM service reliability, you must expand your evaluation gate triggers beyond just prompt or model file changes. Your current "paths: ["prompts/**"]" filters likely miss critical regressions from tokenizer updates, retrieval index rebuilds, or SDK version bumps. Implement a lightweight canary eval across your entire service path and pin provider models to dated snapshots. This proactive approach will catch silent output degradations before they impact users, preventing false confidence from green but incomplete eval gates.

Key insights

Eval gates often miss regressions caused by non-prompt changes, requiring broader coverage and input monitoring.

Principles

• Output quality depends on many non-prompt inputs.
• Gate coverage must match all potential output movers.
• Green eval gates don't guarantee full coverage.

Method

Proposes listing every input affecting output (prompt, tokenizer, retrieval index, tool schemas, SDK version, etc.), then checking if eval runs are triggered and datasets exercise each.

In practice

• Implement a small, fast canary eval on all service merges.
• Pin provider models to dated snapshots, not aliases.
• Monitor alias resolution for unexpected model updates.

Topics

• LLM Evaluation
• Regression Testing
• MLOps Practices
• RAG Systems
• Model Versioning
• CI/CD Workflows

Best for: AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• I Thought Prompt Versioning Was Overkill. Then I Couldn’t Explain Why Production Changed Overnight. — Prompt changes are executable code, requiring version control, testing, and logging to prevent silent production degradation.
• Better Experiments with LLM Evals — A funnel, not a fork — LLM evals verify quality before experiments, which validate user outcomes, forming a crucial calibration funnel.
• A 91% eval pass rate shipped our worst regression. We gate on the delta now. — Absolute evaluation thresholds are insufficient for regression detection; delta-based gating against a stable baseline is crucial.
• The Eval Flywheel: Turning Every Production AI Failure Into a Regression Test — Convert every production AI failure into a permanent regression test to prevent recurrence.
• LLM Model Serving Best Practices: From a Working Endpoint to a Reliable System — Production LLM serving requires optimizing "goodput" and admission control to manage KV-cache pressure and tail latency.
• When Claude changed, everything changed: Managing AI blast radius in production — LLM upgrades can introduce unpredictable "infinite blast radius" failures, necessitating robust evaluation as the true system specification.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by LLM on Medium.