Your First Task as a Data Engineer in a New Company? Make the ETL Pipeline Testable

2026-06-24 · Source: Towards Data Science · Field: Technology & Digital — Data Science & Analytics, Software Development & Engineering · Depth: Intermediate, medium

Summary

A new data engineer inheriting ETL pipelines often faces challenges such as unexpected upstream schema changes, subtle data quality issues, outdated documentation, and performance degradation due to volume growth. This article proposes an automated testing workflow to mitigate these problems. It details setting up a reproducible testing environment using Docker Desktop, VS Code, and the Dev Containers extension, which allows for isolated execution of tests. The workflow involves installing Python, Java, and Poetry, and configuring `.devcontainer` files. The article differentiates between unit tests, which validate small logic components, and integration tests, which verify the entire pipeline's behavior, including data ingestion, transformation, and output. It provides code examples for both. Additionally, it discusses how AI tools like Cursor, Windsurf, and GitHub Copilot can accelerate test generation and code explanation, while emphasizing the continued necessity of human judgment for validating business requirements.

Key takeaway

For data engineers inheriting existing ETL pipelines, prioritizing the implementation of automated testing is critical. This approach, leveraging tools like Docker and VS Code with Dev Containers, ensures pipeline reliability by catching schema changes and data quality issues early. It also provides clear documentation of expected system behavior. While AI tools can accelerate test creation, your expertise remains vital for validating business logic and ensuring robust data integrity.

Key insights

Automated testing is crucial for maintaining inherited ETL pipelines, ensuring data quality and pipeline reliability.

Principles

• Tests define system behavior faster than code.
• Isolated environments ensure consistent testing.
• AI accelerates test generation, not validation.

Method

Set up a reproducible testing environment with Docker, VS Code, and Dev Containers. Implement unit tests for isolated logic and integration tests for end-to-end pipeline validation.

In practice

• Use Docker for isolated test environments.
• Write unit tests for individual functions.
• Implement integration tests for full pipeline validation.

Topics

• ETL Pipeline Testing
• Data Quality
• Docker Containers
• VS Code Extensions
• AI Code Generation
• PySpark

Code references

• company/data-ingestion-transformation

Best for: Data Engineer, MLOps Engineer

Related on AIssential

• Building ETL Pipelines in Databricks | Data Engineering in Databricks — Databricks ETL pipelines transform raw data into production-ready insights using Medallion Architecture and AI-assisted coding.
• Momentic raises the bar for software testing with agentic quality platform — The "AI Paradox" necessitates agentic QA platforms that understand context and classify failures to manage exponentially scaling AI-generated code and bugs.
• I Thought Data Engineering Was Just Writing Scripts. I Was Wrong. — True data engineering extends beyond scripting to build reliable systems through idempotency, persistence, and scheduling.
• From Reactive ETL to Intelligent Data Systems: How AI Is Transforming Cloud Data Pipelines — AI is transforming cloud data pipelines from reactive ETL to self-healing, intelligent systems for reliable enterprise AI.
• Testing SQL Like a Software Engineer: Unit Testing, CI/CD, and Data Quality Automation — Treat SQL like software by implementing version control, unit tests, and automated data quality checks.
• An AI-First Approach to Data Engineering with Lakeflow and Agent Bricks — Databricks Lakeflow integrates AI functions directly into ETL workflows for scalable, context-aware data processing.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards Data Science.