Don’t upgrade your Airflow DB in-place - Sharing lessons from our Airflow 3 migration

2026-02-20 · Source: Data Engineering on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Cloud Computing & IT Infrastructure · Depth: Intermediate, quick

Summary

A recent production migration from Apache Airflow 2.9 to 3.1 revealed that Airflow 3 is a significant rearchitecture, not a simple version upgrade, leading to a multi-day infrastructure project. Key changes include the removal of the webserver, replaced by `airflow api-server` (a FastAPI + React application), necessitating updates to startup scripts, health checks (from `/health` to `/api/v2/monitor/health`), and Dockerfiles. Additionally, the authentication system was completely rewritten, removing Flask-AppBuilder (FAB) and `airflow users create`. Airflow 3 now uses SimpleAuthManager, which reads user credentials from a flat JSON file, requiring this file to be generated at every container boot due to the ephemeral nature of containers.

Key takeaway

For MLOps Engineers planning an Airflow upgrade, recognize that Airflow 3.x is a rearchitecture, not a minor update. You must account for changes to the webserver command, health check endpoints, and a completely rewritten authentication system that relies on flat JSON files for user management. Plan for a multi-day infrastructure project, including script and Dockerfile updates, rather than a simple `pip install --upgrade`.

Key insights

Airflow 3 is a rearchitecture, not an incremental update, requiring significant infrastructure changes.

Principles

• Airflow 3 deprecates Flask-AppBuilder
• Authentication now uses SimpleAuthManager

Method

Airflow 3 replaces the webserver with `airflow api-server` and shifts authentication to a flat JSON file for SimpleAuthManager.

In practice

• Update startup scripts for `airflow api-server`
• Modify health checks to `/api/v2/monitor/health`
• Generate SimpleAuthManager JSON file at container boot

Topics

• Apache Airflow Migration
• Data Orchestration
• MLOps Infrastructure
• API Server Architecture
• Authentication Systems

Best for: MLOps Engineer, Data Engineer, DevOps Engineer

Related on AIssential

• Production Airflow 3.x on Snowpark Container Services: A Complete Reference Architecture — Deploying Airflow 3.x on SPCS requires a multi-service architecture with careful configuration and robust testing.
• DagsHub now supports MLflow 3.x — MLflow 3.x shifts from run-centric to model-centric tracking, improving GenAI observability and evaluation.
• Slack Eliminates SSH in EMR Pipelines, Migrates 700+ Jobs to Rest-Based Architecture — Replacing direct SSH access with a REST-driven orchestration layer significantly enhances data pipeline security, reliability, and observability.
• Self-Hosting Airflow at Home: Automating Stock Price Data Collection — Self-hosting Airflow enables robust, automated data pipelines for MLOps, even on a homelab.
• How to move from Apache Airflow® to Databricks Lakeflow Jobs — Lakeflow Jobs prioritizes data-driven, event-based orchestration, diverging from Airflow's control-plane-centric model.
• Apache Airflow Document Ingestion Pipeline for RAG Systems — Production RAG ingestion requires robust orchestration, idempotency, and observability for reliable document processing.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Data Engineering on Medium.