Data Quality Diagnostics

2026-03-13 · Source: Data Engineering on Medium · Field: Technology & Digital — Data Science & Analytics, Artificial Intelligence & Machine Learning · Depth: Intermediate, medium

Summary

This article introduces a diagnostic framework for identifying subtle data quality issues that evade standard profiling tools like `.describe()`. It emphasizes that data often appears "plausible" rather than overtly "dirty," leading to silent failures in data pipelines and models. The framework is structured around five key questions: "Does the data match its contract?" (verifying structure, types, and column names), "Is the absence informative?" (analyzing patterns and correlations of missing data), "Are your identifiers trustworthy?" (detecting exact and partial duplicates, and assessing cardinality), "Do the numbers make domain sense?" (identifying outliers and distributional anomalies with domain context), and "Do the fields tell the same story?" (uncovering logical contradictions and referential integrity issues). The article provides minimal `pandas` and `numpy` code snippets to illustrate each diagnostic question, stressing that the quality of the question is more important than the sophistication of the tool.

Key takeaway

For Data Scientists and Machine Learning Engineers building robust data pipelines, you should move beyond basic data profiling. Actively ask diagnostic questions about your data's contract, missingness patterns, identifier integrity, numerical domain sense, and cross-field consistency. This proactive, question-driven approach will uncover silent data quality issues before they compromise model accuracy or downstream analysis, saving significant debugging time.

Key insights

Subtle data quality issues often manifest as "plausible" data, requiring diagnostic questions beyond basic profiling.

Principles

• Data quality is a continuous monitoring process, not a one-time check.
• The meaning of data anomalies depends on domain knowledge.
• Assertions are hypotheses about data, not just safeguards.

Method

A diagnostic approach to data quality involves asking five key questions: contract adherence, informative absence, identifier trustworthiness, domain sense of numbers, and cross-field consistency, using basic Python tools.

In practice

• Use `df.select_dtypes(include="object").apply(lambda col: pd.to_numeric(col, errors="coerce").notna().sum())` to find numeric strings.
• Visualize missingness patterns with `df.isnull().T` to reveal structural issues.
• Assert `df["id"].is_unique` to explicitly test identifier uniqueness.

Topics

• Data Quality
• Data Diagnostics
• Data Validation
• Missing Data Analysis
• Data Profiling

Best for: Data Scientist, Data Engineer, Machine Learning Engineer

Related on AIssential

• Bad Data Will Find You: Validation Patterns for Streaming Pipelines — Robust data validation is critical for streaming pipelines to prevent silent corruption and state pollution from inevitable bad data.
• 5 Useful Python Scripts for Advanced Data Validation & Quality Checks — Advanced data validation requires automated scripts to detect subtle semantic, temporal, and structural data quality issues.
• Never seen a data quality issue that wasn’t actually an ownership problem | John Wernfeldt — Data quality issues are primarily ownership problems, not technical ones, hindering AI readiness.
• Probabilistic data quality assessment for structural monitoring data via outlier-resistant conditional diffusion model — A conditional diffusion model improves SHM data quality assessment by robustly identifying and quantifying outliers.
• The 8 Lineage Gaps That Make ML Bugs Untraceable — Untraceable ML bugs often stem from incomplete data lineage in critical data transformation steps.
• How Agoda Built a Single Source of Truth for Financial Data — Consolidating disparate data pipelines into a unified system enhances data consistency and operational efficiency.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Data Engineering on Medium.