Is this data actually good, or does it just look good?

2026-06-28 · Source: Alex Strick van Linschoten · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, medium

Summary

Alex Strick van Linschoten audited an ISAF press-release dataset, intended for RL model training and evaluation, uncovering significant data quality issues. The audit revealed misspelled province names, such as "paktya" instead of "paktia" across 197 entries, leading to 38 distinct province values instead of the expected 34. Other problems included ambiguous region descriptions like "southern Afghanistan" and multiple province assignments ("khost;paktya"). The author categorized fixes into mechanical, judgement-based, and caution tiers. A critical finding was a subtle train/test leak, where identical press releases appeared in both splits. To address these, the author implemented non-destructive cleaning by adding `province_clean` and `province_status` columns, tagged the original dataset as `v1`, and updated the datacard. Hashing input/output pairs was also introduced to maintain train/test split integrity.

Key takeaway

For Data Scientists or ML Engineers preparing datasets for model training, you must conduct thorough data quality audits beyond surface-level checks. Prioritize non-destructive cleaning by adding new columns for corrected values and status, preserving an audit trail. This prevents silent inaccuracy inflation and misplaced confidence, ensuring your model learns from genuinely reliable data and avoids issues like train/test leakage.

Key insights

Data quality audits are crucial for machine learning, revealing hidden issues like misspellings, ambiguity, and train/test leakage that impact model performance.

Principles

• Separate data fixes by type: mechanical, judgement, caution.
• "Unknown" is a valid, honest data value.
• Preserve original data via non-destructive cleaning.

Method

Audit gold labels, categorize data quality issues, apply non-destructive fixes by adding new columns for cleaned values and status, tag dataset versions, and hash input/output pairs to prevent train/test leakage.

In practice

• Use lookup tables for mechanical data corrections.
• Assign "unknown" for genuinely ambiguous data points.
• Create `_clean` and `_status` columns for audit trails.

Topics

• Data Quality
• Dataset Auditing
• Reinforcement Learning
• Train/Test Split
• Data Cleaning
• Gold Labels

Best for: Machine Learning Engineer, Data Scientist, AI Student

Related on AIssential

• Data Cleaning Techniques for Improved AI Predictions. — Effective data cleaning is fundamental for transforming raw data into reliable inputs for accurate analysis and machine learning models.
• Why Your 99% Accurate Model Might Actually Be Useless — Data leakage inflates model accuracy by providing unavailable information, leading to real-world failure.
• Tools to Improve Training Data — Data quality issues can render high model accuracy meaningless, necessitating human-in-the-loop tools for robust ML.
• Why Data Cleaning Takes Longer Than the Model In AI — High-quality data preparation is more critical for machine learning success than complex algorithms.
• 7 myths about “more data” (and why models get worse) — More data can degrade model performance by introducing noise, distribution shifts, or label poisoning.
• Common Data Science Mistakes and How to Avoid Them — Common data science mistakes often produce plausible but incorrect results, requiring deliberate checks.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Alex Strick van Linschoten.