I built an AI data investigator that refuses to make up numbers

2026-06-21 · Source: Data Science on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, medium

Summary

Veritas is an open-source MCP server designed to prevent AI tools, specifically Claude, from generating fabricated numerical data in analytical reports. It operates on a "no receipt, no claim" principle, where every number in a final answer must trace back to a verified SQL query or Python script result. The system treats complex questions as research projects, generating competing explanations and using deterministic Python code to validate all numerical claims against saved "receipts." Veritas also features a discovery mode that aggressively filters potential findings using Benjamini-Hochberg false-discovery-rate control and an effect-size floor, prioritizing silence over statistical noise. It is deliberately narrow, working with local CSV, Parquet, and Excel files via DuckDB, and includes a public evaluation suite for rigor.

Key takeaway

For data scientists or AI engineers building analytical agents, Veritas offers a critical blueprint for ensuring numerical accuracy and trustworthiness. You should integrate deterministic verification steps, like those in Veritas, into your AI-driven data investigation workflows to prevent hallucinated numbers. Consider adopting its "receipt" system and aggressive filtering for discovery modes to build more reliable and accountable AI tools, especially when dealing with sensitive business metrics.

Key insights

AI data investigation must enforce numerical veracity through verifiable query results, not just plausible language.

Principles

• Every number needs a verifiable source.
• Deterministic verification prevents AI self-grading.
• Aggressive suppression improves signal-to-noise.

Method

Claude splits questions into hypotheses, runs queries, and saves results as artifacts. A deterministic Python step then verifies each reported number against these artifacts.

In practice

• Integrate deterministic verification into AI data pipelines.
• Use false-discovery-rate control for AI-driven insights.
• Leverage DuckDB for local data investigation.

Topics

• AI Data Investigation
• Hallucination Prevention
• Deterministic Verification
• MCP Server
• Claude Integration
• DuckDB

Code references

• advait27/Veritas

Best for: AI Architect, Machine Learning Engineer, AI Product Manager, AI Engineer, MLOps Engineer, Data Scientist

Related on AIssential

• The Silent Failure Problem — How an Empty Result Almost Fooled Our Agent — Silent failures, where systems return incorrect empty results without error, are more dangerous than loud failures.
• How Kepler built verifiable AI for financial services with Claude — Combining AI reasoning with deterministic infrastructure enables verifiable, auditable financial analysis.
• AI at college graduations and why Claude blackmails — AI's rapid advancement elicits public apprehension, while its practical application demands human oversight and high-quality data.
• Phantoms and Disclosures: a Causal Framework for Auditing Synthetic Data — The framework distinguishes true from phantom data disclosures in synthetic data using statistical testing, requiring no model access.
• Pramaana Labs raises $27M seed round from Khosla Ventures to bring formal verification to AI — Combining LLMs with formal verification offers a path to reliable AI in sensitive domains by codifying rules.
• AI Hallucinations Are a Verification Issue — Since AI hallucinations are unavoidable, prioritize robust output verification over attempting to eliminate them entirely.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Data Science on Medium.