I Thought Data Engineering Was Just Writing Scripts. I Was Wrong.

2026-06-12 · Source: Towards Data Science · Field: Technology & Digital — Data Science & Analytics, Software Development & Engineering · Depth: Novice, medium

Summary

The article describes a data analyst's journey to understand data engineering beyond simple scripting, detailing the evolution of an ETL pipeline built using the GitHub API. Initially, the pipeline extracted 30 Python repositories created after 2025-04-22, cleaned descriptions, and saved them to a CSV file. The author encountered three significant challenges when attempting to make the pipeline "production-ready": lack of memory leading to duplicate data, data disappearance due to non-persistence, and the need for manual execution. Upgrades involved switching from CSV to a SQLite database, implementing idempotency to prevent duplicates by deleting and re-inserting records based on URL, and persisting the SQLite database to Google Drive to ensure data survival across sessions. The next identified challenge is automated scheduling using tools like Apache Airflow or Prefect, highlighting that data engineering focuses on building reliable systems, not just one-off scripts.

Key takeaway

For data analysts transitioning to data engineering, your initial ETL scripts, while functional, will likely fail in production environments. You must prioritize building reliable systems by addressing idempotency to prevent data duplication, ensuring data persistence beyond single sessions, and implementing automated scheduling. Neglecting these aspects turns a pipeline into a liability, risking data integrity and operational continuity. Start integrating these engineering principles early in your development process.

Key insights

True data engineering extends beyond scripting to build reliable systems through idempotency, persistence, and scheduling.

Principles

• Pipelines need idempotency to prevent duplicates.
• Data persistence is crucial for system reliability.
• Automated scheduling ensures continuous operation.

Method

An ETL pipeline was upgraded by replacing CSV output with SQLite, implementing a delete-then-insert strategy for idempotency, and persisting the database to Google Drive for session-independent data storage.

In practice

• Use SQLite instead of CSV for queryable data.
• Implement delete-then-insert for idempotency.
• Store databases in persistent cloud storage.

Topics

• Data Engineering
• ETL Pipelines
• Idempotency
• Data Persistence
• Pipeline Scheduling
• SQLite
• GitHub API

Best for: Data Analyst, Data Engineer, AI Student

Related on AIssential

• Why the Best Data Engineers Never Memorize Tools. — Mastering data engineering patterns, not just tools, ensures adaptability and robust pipeline design.
• Job Orchestration in Databricks | Data Engineering in Databricks — Databricks Jobs automate ETL pipelines with flexible task orchestration and diverse triggering mechanisms.
• How to Build a Data Warehouse (Full Lifecycle Explained) — A robust data warehouse lifecycle ensures business alignment, data integrity, and continuous operational resilience.
• Your Data Lives Everywhere, and That is the Problem — Data integration is the core task in data engineering due to data scattering across diverse systems.
• I Built My First ETL Pipeline as a Complete Beginner. Here’s How. — Building teaches more than consuming theory, especially for data engineering fundamentals.
• Eventual Consistency: The Key Trade-Off Behind Modern Databases — Eventual consistency trades immediate data synchronization for enhanced performance, scalability, and availability in distributed systems.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards Data Science.