Run an Apache Airflow DAG with Docker Compose and PostgreSQL

2026-06-08 · Source: PyImageSearch · Field: Technology & Digital — Software Development & Engineering, Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, extended

Summary

This lesson details the operationalization of a production-grade document ingestion pipeline, integrating Apache Airflow, FastAPI, and PostgreSQL within a Docker Compose environment. Building on a previous architectural design for RAG systems, it guides users through setting up the entire system, including project structure, PDF parsing, and text chunking logic. The article demonstrates running the pipeline using "docker compose up --build", uploading documents via a FastAPI endpoint at "http://localhost:8000/docs", and monitoring DAG executions in the Airflow UI at "http://localhost:8080" (login: "admin"/"admin"). It covers verifying processed data in PostgreSQL, handling corrupted PDF failure scenarios, and highlights design principles such as idempotency, observability, and reproducibility. Finally, it discusses the practical limitations of Apache Airflow for GPU-accelerated or massively parallel machine learning workloads.

Key takeaway

For MLOps Engineers designing document ingestion pipelines, this Docker Compose and Airflow setup provides a robust, observable, and idempotent framework. You should adopt shared volumes for inter-service file access and implement content hashing for deduplication to prevent data corruption and ensure reliable processing. Consider Airflow for ETL and scheduled tasks, but plan for alternative orchestrators like Argo Workflows for GPU-accelerated or highly parallel ML workloads.

Key insights

Operationalizing data pipelines with Docker Compose ensures reproducible, observable, and idempotent document ingestion for RAG systems.

Principles

• Separate ingestion from heavy processing for scalability.
• Implement idempotency via content hashing and status tracking.
• Ensure observability through comprehensive logging and metrics.

Method

Orchestrate Airflow, FastAPI, and PostgreSQL using Docker Compose. Mount shared code and a common upload volume. Define idempotent DAGs for PDF parsing, chunking, and database updates.

In practice

• Use "docker compose up --build" for consistent deployment.
• Mount shared code ("shared/") across services for consistency.
• Implement SHA-256 hashing for document deduplication.

Topics

• Apache Airflow
• Docker Compose
• Document Ingestion Pipeline
• RAG Systems
• PostgreSQL Database
• FastAPI Service
• Idempotent Pipelines

Best for: MLOps Engineer, AI Engineer, Data Engineer

Related on AIssential

• Apache Airflow Document Ingestion Pipeline for RAG Systems — Production RAG ingestion requires robust orchestration, idempotency, and observability for reliable document processing.
• Understanding Apache Airflow: DAGs, Tasks, and Operators Explained Simply — Airflow orchestrates workflows using DAGs as blueprints, Tasks as work units, and Operators as execution templates.
• How to move from Apache Airflow® to Databricks Lakeflow Jobs — Lakeflow Jobs prioritizes data-driven, event-based orchestration, diverging from Airflow's control-plane-centric model.
• 5 Docker Containers for Small Business — Containerization with Docker provides small businesses an affordable, integrated data infrastructure solution.
• DataOps with Dagster: A Practical Guide to Building a Reliable Data Platform — DataOps integrates development and operations principles to build trustworthy, visible, and controllable data systems.
• Production Airflow 3.x on Snowpark Container Services: A Complete Reference Architecture — Deploying Airflow 3.x on SPCS requires a multi-service architecture with careful configuration and robust testing.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by PyImageSearch.