I Built My Second ETL Pipeline. This Time, I Started Thinking Like a Data Engineer

2026-07-10 · Source: Towards Data Science · Field: Technology & Digital — Data Science & Analytics, Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Novice, long

Summary

The author built an automated RSS ingestion pipeline to explore data engineering principles, emphasizing the separation of concerns among Python for business logic, PostgreSQL for idempotent data storage (`ON CONFLICT DO NOTHING`), Docker for consistent execution, and Kestra for scheduling and orchestration. This second ETL project, following a GitHub ETL pipeline, focused on transforming a simple script into a reliable system. Key decisions included validating each layer incrementally (ETL, DB, Docker, Kestra) to simplify debugging, packaging the application as a Docker image for consistent runtime, and configuring Kestra for automated hourly schedules and retries (e.g., `maxAttempts: 3`, `interval: PT30S`). The project highlighted that engineering effort often centers on reliability and system design rather than just ETL logic.

Key takeaway

For data engineers building automated pipelines, prioritize architectural separation of concerns over complex ETL logic. You should validate each component—application, database, and container—independently before integrating with orchestrators like Kestra. This approach ensures reliability, simplifies debugging, and transforms basic scripts into robust, production-ready systems by clearly defining responsibilities for execution, persistence, and orchestration.

Key insights

Reliable data pipelines require clear separation of responsibilities among application logic, execution environment, and orchestration.

Principles

• Validate each layer incrementally.
• Separate execution from orchestration.
• Each component should have a single responsibility.

Method

Build data pipelines incrementally: validate Python ETL, then PostgreSQL persistence, then Docker image, finally orchestrate with Kestra, ensuring each layer works independently.

In practice

• Use `ON CONFLICT DO NOTHING` for idempotent inserts.
• Containerize ETL with Docker for consistent execution.
• Configure Kestra for retries (e.g., `maxAttempts: 3`, `interval: PT30S`).

Topics

• Data Engineering
• ETL Pipelines
• Workflow Orchestration
• Docker Containerization
• Idempotent Data Processing
• PostgreSQL
• System Design

Best for: Data Engineer, MLOps Engineer, AI Student

Related on AIssential

• I Thought Data Engineering Was Just Writing Scripts. I Was Wrong. — True data engineering extends beyond scripting to build reliable systems through idempotency, persistence, and scheduling.
• Your First Task as a Data Engineer in a New Company? Make the ETL Pipeline Testable — Automated testing is crucial for maintaining inherited ETL pipelines, ensuring data quality and pipeline reliability.
• Adding Humans to Your Data Pipelines: Orchestra’s Approval Integration — Orchestra's approval integration provides human-in-the-loop control for critical data pipeline operations, balancing autonomy with guardrails.
• Building ETL Pipelines in Databricks | Data Engineering in Databricks — Databricks ETL pipelines transform raw data into production-ready insights using Medallion Architecture and AI-assisted coding.
• From Stock Prices to Sales Performance: Orchestrating the Platform— Part 6 — Effective data pipeline orchestration prioritizes modularity and explicit dependency management over monolithic designs.
• Main Data Warehouse Components Explained (2025) — Data warehouses organize vast datasets into actionable insights using a layered architecture and dedicated processing.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards Data Science.