Self-Hosting Airflow at Home: Automating Stock Price Data Collection

2026-06-13 · Source: Data Engineering on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Software Development & Engineering · Depth: Intermediate, long

Summary

The article details setting up a robust Apache Airflow instance in a homelab for MLOps, specifically to automate stock price data collection into a PostgreSQL database. It outlines configuring Airflow components like the scheduler, DAG processor, triggerer, and API server as systemd daemon processes for continuous, resilient operation, including a script for restarting all components. The author explains how to establish a PostgreSQL connection within the Airflow UI and then presents Python code for a daily DAG. This DAG uses the "yfinance" package to fetch 5-year historical Canadian stock data from a CSV watchlist, transforms it with pandas, and writes it to a "finance.watchlist_cad_ticker_price" table. The article also covers deploying DAGs using a GitHub repository for version control and discusses future improvements such as local testing, Docker for environment management, and alerting.

Key takeaway

For MLOps Engineers or Data Engineers building automated data pipelines in a homelab, this guide demonstrates a robust, self-hosted Airflow setup. You can ensure continuous operation by configuring Airflow components as systemd daemons, preventing crashes. Deploying DAGs via a GitHub repository streamlines version control and updates, mirroring production practices. This approach provides a resilient and scalable data foundation for training machine learning models without external cloud dependencies.

Key insights

Self-hosting Airflow and PostgreSQL automates financial data collection for MLOps workflows with enhanced resilience and version control.

Principles

• Use systemd for Airflow component resilience.
• Version control DAGs via Git for deployment.
• Decouple Airflow environments with Docker.

Method

Configure Airflow components as systemd daemons, establish PostgreSQL connection in Airflow UI, then develop and deploy Python DAGs via Git for automated data ingestion.

In practice

• Set up Airflow components as systemd services.
• Use "yfinance" to fetch market data.
• Manage DAG code with a GitHub repository.

Topics

• Apache Airflow
• MLOps
• Data Pipelines
• PostgreSQL
• Systemd
• yfinance
• Homelab

Code references

• miketcchung/homelab-data-engineering
• miketcchung/homelab-data-engineering

Best for: MLOps Engineer, Data Engineer, AI Student

Related on AIssential

• Self-Hosting Airflow at Home: Automating Stock Price Data Collection — Self-hosting Airflow enables robust, automated data pipelines for MLOps, even on a homelab.
• Run an Apache Airflow DAG with Docker Compose and PostgreSQL — Operationalizing data pipelines with Docker Compose ensures reproducible, observable, and idempotent document ingestion for RAG systems.
• How to move from Apache Airflow® to Databricks Lakeflow Jobs — Lakeflow Jobs prioritizes data-driven, event-based orchestration, diverging from Airflow's control-plane-centric model.
• Scalable ML Runtime Deployment with H2O MLOps | Part 7 — Automated validation and isolated, scalable runtime environments are crucial for robust ML model deployment.
• Apache Kafka for Data Engineers — Beginner Guide — Apache Kafka provides a scalable, fault-tolerant, and decoupled architecture for real-time data streaming.
• DataOps with Dagster: A Practical Guide to Building a Reliable Data Platform — DataOps integrates development and operations principles to build trustworthy, visible, and controllable data systems.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Data Engineering on Medium.