I Tried to Schedule My ETL Pipeline. Here’s What I Didn’t Expect.

2026-06-19 · Source: Towards Data Science · Field: Technology & Digital — Data Science & Analytics, Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Novice, medium

Summary

A systems analyst transitioning into data engineering discovered that making an ETL pipeline portable was a critical prerequisite to scheduling it, a challenge unexpected before selecting a scheduling tool. The author's existing pipeline, built in Google Colab, was tied to the platform by a hardcoded database path (`/content/drive/MyDrive/`). The solution involved externalizing this path using an environment variable, allowing the `pipeline.py` script to run independently. Subsequently, GitHub Actions was chosen for scheduling due to its free, serverless nature, despite its limitations compared to full orchestration tools like Airflow. A YAML workflow (`schedule.yml`) was configured to run the pipeline daily at 9am UTC on an `ubuntu-latest` runner, demonstrating successful automated execution and highlighting the distinction between scheduling and orchestration.

Key takeaway

For Data Engineers building or migrating ETL pipelines, prioritize making your scripts environment-agnostic before attempting scheduling. Hardcoded paths or platform-specific dependencies will prevent automation. You should externalize such configurations using environment variables. This ensures your pipeline can run anywhere, from a local machine to a cloud-based scheduler like GitHub Actions, which handles basic cron-based execution effectively. Understand that simple scheduling tools differ from full orchestration platforms like Airflow.

Key insights

Pipeline portability is a prerequisite for effective scheduling and robust data engineering.

Principles

• Environment is part of the pipeline.
• Hardcoded paths create platform dependencies.
• Scheduling differs from orchestration.

Method

To schedule an ETL pipeline, first ensure portability by externalizing environment-specific dependencies like hardcoded paths. Then, define a GitHub Actions workflow with cron scheduling and necessary setup steps.

In practice

• Externalize database paths using `os.environ.get('DB_PATH', 'default.db')`.
• Schedule daily runs at 9am UTC with `cron: '0 9 * * *'` in GitHub Actions.
• Update GitHub Actions to `actions/checkout@v4` and `actions/setup-python@v5`.

Topics

• ETL Pipeline
• GitHub Actions
• Data Engineering
• Pipeline Scheduling
• Portability
• Google Colab

Best for: Data Engineer, AI Student, MLOps Engineer

Related on AIssential

• I Thought Data Engineering Was Just Writing Scripts. I Was Wrong. — True data engineering extends beyond scripting to build reliable systems through idempotency, persistence, and scheduling.
• Day 23: Scheduling Deeper, Cron Syntax, Timer Units, and One-Time Jobs — This article delves into advanced Linux task scheduling, detailing cron syntax, systemd timer units, and the "at" command for one-time jobs.
• Understanding Apache Airflow: DAGs, Tasks, and Operators Explained Simply — Airflow orchestrates workflows using DAGs as blueprints, Tasks as work units, and Operators as execution templates.
• Job Orchestration in Databricks | Data Engineering in Databricks — Databricks Jobs automate ETL pipelines with flexible task orchestration and diverse triggering mechanisms.
• Slack Eliminates SSH in EMR Pipelines, Migrates 700+ Jobs to Rest-Based Architecture — Replacing direct SSH access with a REST-driven orchestration layer significantly enhances data pipeline security, reliability, and observability.
• I Built My First ETL Pipeline as a Complete Beginner. Here’s How. — Building teaches more than consuming theory, especially for data engineering fundamentals.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards Data Science.