We Shipped an AI That Confidently Lied About Websites. Here’s How Evals Fixed It.

2026-07-12 · Source: LLM on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, long

Summary

DoableClaw's AI audit engine, CLAW, initially produced "hallucinated absence claims," confidently reporting missing website features like pricing pages when its Playwright-based crawler failed to retrieve data due to issues like JavaScript walls or bot-blocking. This critical failure mode, which nearly eroded trust, was traced to pipeline problems rather than the LLM itself, as the model merely laundered insufficient evidence into fluent prose. To counter this, DoableClaw implemented a robust LLM evaluation (evals) system. This involved creating a golden dataset from real production failures, enforcing a "cite-or-drop" rule for all load-bearing claims, and applying a special rule for absence claims to only assert absence with sufficient crawl evidence. They also introduced human-in-the-loop TRUE/BUSTED scorecards and conducted multi-model bake-offs, discovering that the most expensive model was not always the best value for their specific tasks. The system now uses a multi-model parallel architecture, with different models chosen by measured fit.

Key takeaway

For MLOps Engineers building LLM-powered intelligence systems, prioritize robust evaluation of your entire pipeline, not just the model output. Your system's fluency can mask upstream data retrieval failures, leading to "hallucinated absence claims" that erode user trust. Implement "cite-or-drop" gates and specific rules for negative assertions, and use real production failures to build your evaluation datasets. This approach will measurably improve output accuracy and help you select cost-effective models.

Key insights

LLM fluency doesn't equal accuracy; robust evaluation, especially of upstream data pipelines, is crucial for trustworthy AI systems.

Principles

• LLM fluency does not imply accuracy.
• Systemic pipeline issues often manifest as model hallucinations.
• Ground all critical claims in primary sources.

Method

Build golden datasets from real failures. Implement "cite-or-drop" gates and special absence claim rules. Use human-calibrated LLM judges and multi-model bake-offs to measure and optimize quality.

In practice

• Create golden sets from actual production errors.
• Enforce "cite-or-drop" for all critical claims.
• Calibrate LLM judges against human review.

Topics

• LLM Evaluation
• Hallucination Detection
• RAG Metrics
• Pipeline Evals
• Golden Datasets
• Multi-model Architectures

Code references

• danielrosehill/Awesome-AI-Evaluations-Tools

Best for: AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Every AI output is a hallucination — Large Language Models inherently produce fluent, confident outputs that are often wrong yet indistinguishable from truth, posing significant reliability issues.
• The Poisoned Chalice of LLM Evaluation Report — Data contamination in LLM evaluation for software engineering requires robust, generalizable white-box membership inference methods.
• Your LLM Is Only as Good as What It Retrieves — Retrieval quality, not model scale, is the primary determinant of LLM factual accuracy and hallucination prevention in RAG systems.
• Building an Evaluation Harness for Production AI Agents: A 12-Metric Framework From 100+ Deployments — Robust evaluation infrastructure is critical for successful, compliant AI agent deployments in production.
• From Wordle to Fibble5: Evaluating LLM Reasoning Under Escalating Deception — LLM reasoning robustness is severely degraded by even minimal deception, a factor poorly captured by standard benchmarks.
• Dont ignore omissions! — LLM omissions are a critical, under-researched problem, especially in high-stakes domains like medicine.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by LLM on Medium.