Production Airflow 3.x on Snowpark Container Services: A Complete Reference Architecture

2026-03-13 · Source: Data Engineering on Medium · Field: Technology & Digital — Cloud Computing & IT Infrastructure, Data Science & Analytics, Artificial Intelligence & Machine Learning · Depth: Intermediate, medium

Summary

This article details a production-grade reference architecture for deploying Apache Airflow 3.1.7 on Snowpark Container Services (SPCS), addressing the complexities of running seven interdependent services across three compute pools. The architecture leverages SPCS for data governance, network, and billing integration, utilizing an INFRA_POOL for PostgreSQL 17.9 and Redis 7.4, a CORE_POOL for Airflow's brain services (API server, scheduler, DAG processor, triggerer), and an auto-scaling WORKER_POOL for CeleryExecutor tasks. The solution includes 542 automated tests to prevent configuration drift and highlights 13 specific SPCS "gotchas" related to DNS, secret injection, readiness probes, and volume permissions, providing a robust, open-source framework for Airflow orchestration within Snowflake.

Key takeaway

For MLOps Engineers or AI Architects considering Airflow orchestration within Snowflake, this reference architecture provides a validated path. You should adopt the multi-pool service deployment and leverage Snowflake SECRET objects for critical configurations like JWTs. Pay close attention to the 13 SPCS-specific "gotchas" to avoid silent failures, especially regarding DNS naming, secret injection, and the `dag-processor` service. Utilize the provided three-tier test suite to prevent configuration drift and ensure deployment reliability.

Key insights

Deploying Airflow 3.x on SPCS requires a multi-service architecture with careful configuration and robust testing.

Principles

• Isolate services into distinct compute pools for performance.
• Use Snowflake SECRET objects for consistent JWT secret sharing.
• Validate configurations with comprehensive static and runtime tests.

Method

The proposed method involves deploying seven Airflow services across three SPCS compute pools, managing secrets via Snowflake SECRET objects, and validating the entire stack with a three-tier test suite (offline, local integration, end-to-end).

In practice

• Configure `AIRFLOW__CORE__EXECUTION_API_SERVER_URL` with `/execution/` suffix.
• Run `dag-processor` as a separate SPCS service.
• Set `AIRFLOW__SCHEDULER__SYMLINK_LATEST_LOG=False` for GCP stage mounts.

Topics

• Snowpark Container Services
• Apache Airflow 3.x
• Data Pipeline Orchestration
• Reference Architectures
• Configuration Validation

Code references

• sfc-gh-jkang/spcs-airflow

Best for: MLOps Engineer, Data Engineer, AI Architect

Related on AIssential

• Self-Hosting Airflow at Home: Automating Stock Price Data Collection — Self-hosting Airflow and PostgreSQL automates financial data collection for MLOps workflows with enhanced resilience and version control.
• Building a Production-Ready Snowflake MCP Server — Building a secure Snowflake MCP server involves specific deployment, authentication, and SQL patterns.
• Don’t upgrade your Airflow DB in-place - Sharing lessons from our Airflow 3 migration — Airflow 3 is a rearchitecture, not an incremental update, requiring significant infrastructure changes.
• Building a European Cloud Orchestration Platform within an Enterprise — Kubernetes Control Planes unify multi-cloud resource orchestration, reducing engineering burden and manual operations.
• Deploying Existing Agentic Services to Microsoft 365 Copilot via Gateway and OBO — Deploy existing agentic services to M365 Copilot by using a stateless gateway and chained On-Behalf-Of token exchanges for user-delegated access.
• Building a fault-tolerant metrics storage system at Airbnb — Scaling observability requires robust multi-tenant isolation, automated management, and strategic multi-cluster deployment to ensure reliability.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Data Engineering on Medium.