How to move from Apache Airflow® to Databricks Lakeflow Jobs

2026-03-18 · Source: Databricks · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Software Development & Engineering · Depth: Intermediate, medium

Summary

Databricks Lakeflow Jobs offers a distinct orchestration paradigm compared to Apache Airflow, emphasizing data-driven coordination over control-plane signals. Key assumptions include separating control plane from data plane operations, treating jobs as the primary unit of orchestration with data (tables, files) for cross-job communication, and integrating first-class file arrival and table update triggers. This approach shifts designs from "DAG talking to DAG" to a producer-consumer model where jobs trigger on data changes. The migration guide details how to transition Airflow patterns like XComs, sensors, execution dates, branching, and dynamic mapping to Lakeflow's task values, file/table triggers, explicit parameters, and condition/for-each tasks, respectively. It also introduces Python Asset Bundles for programmatic job generation.

Key takeaway

For MLOps Engineers or Data Engineers migrating from Airflow to Databricks, internalizing Lakeflow's data-first orchestration model is crucial. Focus on replacing polling-based sensors with event-driven file and table triggers, and refactor XComs into task values for control flow and Unity Catalog tables for data. This shift will simplify pipeline design and improve efficiency by leveraging the lakehouse as the shared state, rather than relying on complex cross-DAG signaling.

Key insights

Lakeflow Jobs prioritizes data-driven, event-based orchestration, diverging from Airflow's control-plane-centric model.

Principles

• Data plane operations drive compute usage.
• Jobs coordinate via data, not cross-DAG signals.
• Triggers are first-class, event-driven features.

Method

Migrate Airflow XComs to Lakeflow task values for control metadata and Unity Catalog tables for data. Replace sensors with file/table triggers. Model execution dates as explicit parameters for backfills. Convert branching and dynamic mapping to condition and for-each tasks.

In practice

• Use task values for small control metadata (flags, IDs).
• Store large data payloads in Unity Catalog tables.
• Trigger jobs on file arrivals or table updates.

Topics

• Lakeflow Jobs
• Airflow Migration
• Data Orchestration
• Unity Catalog
• Event-Driven Triggers

Code references

• databricks/bundle-examples

Best for: Data Engineer, MLOps Engineer, Machine Learning Engineer

Related on AIssential

• Job Orchestration in Databricks | Data Engineering in Databricks — Databricks Jobs automate ETL pipelines with flexible task orchestration and diverse triggering mechanisms.
• Lakeflow: A new era of agentic data engineering — Databricks Lakeflow unifies data engineering with AI agents for pipeline development, operations, and real-time ingestion.
• Run an Apache Airflow DAG with Docker Compose and PostgreSQL — Operationalizing data pipelines with Docker Compose ensures reproducible, observable, and idempotent document ingestion for RAG systems.
• How to Orchestrate Across Multiple Databricks Workspaces Without Losing Your Mind — Managing cross-workspace dependencies is critical for scalable Databricks orchestration, which native tools lack.
• Open Platform, Unified Pipelines: Why dbt on Databricks is Accelerating — Consolidating dbt workflows on an open lakehouse platform enhances data governance, performance, and operational simplicity.
• From Data Analyst to Data Engineer: My 12-Month Self-Study Roadmap — Transitioning to data engineering offers deeper infrastructure understanding and career resilience against AI automation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Databricks.