SAP Data Doesn’t Care About Your Medallion (And I Built One Anyway)

2026-07-27 · Source: Data Engineering on Medium · Field: Technology & Digital — Data Science & Analytics, Artificial Intelligence & Machine Learning · Depth: Intermediate, long

Summary

This article details the implementation of a Medallion architecture for processing complex SAP data within Databricks, utilizing Lakeflow Declarative Pipelines. It addresses common challenges with raw SAP data, such as non-standard date formats like CHAR(16) and "0000-00-00" values, FLOAT types for monetary amounts, and obscure column names like VBELN. The architecture comprises a bronze layer, automatically synced by OneConnect with Change Data Feed enabled; a silver layer that performs Change Data Capture (CDC) to maintain Type-1 snapshots and correctly propagate deletes; and a gold layer that enriches data with text tables, derives calculated fields, and renames technical terms to business-friendly language. A crucial design choice involves separating the generic silver pipeline from domain-specific gold pipelines to ensure data governance and prevent pipeline collisions.

Key takeaway

For data engineers building robust data platforms with SAP sources, you should adopt a multi-pipeline Medallion architecture. This approach, using Lakeflow Declarative Pipelines, ensures proper handling of SAP's complex data types and critical delete propagation. Separate your generic silver layer from domain-specific gold layers to prevent conflicts and maintain data integrity. This strategy provides clean, governed data essential for trustworthy dashboards and advanced AI applications like text-to-SQL agents.

Key insights

Taming chaotic SAP data requires a declarative pipeline approach with careful handling of data types and deletes.

Principles

• Declarative pipelines excel with messy sources like SAP.
• Design for deletes from day one in SAP data pipelines.
• Separate generic shared data from domain-specific pipelines.

Method

Implement a Medallion architecture: Bronze (raw, CDC-enabled), Silver (Type-1 snapshot, apply_as_deletes for CDC), Gold (enrichment, renaming, type casting). Use two pipelines: one for silver, multiple for gold.

In practice

• Intercept "0000-00-00" dates, convert to NULL.
• Cast FLOAT amounts to DECIMAL for financial calculations.
• Filter SAP text tables by language (e.g., spras = 'E').

Topics

• SAP Data Integration
• Medallion Architecture
• Lakeflow Declarative Pipelines
• Change Data Capture
• Databricks Delta Lake
• Data Governance

Code references

• Onibex/agentic-semantic-knowledge-ask

Best for: Data Engineer, MLOps Engineer, AI Architect

Related on AIssential

• A Practical Framework for Designing Scalable Data Pipelines (8 Steps) — Effective data pipeline design requires a structured, business-centric approach from source to consumption.
• What Really Happens Between Bronze, Silver, and Gold Layers? — Medallion Architecture defines clear responsibilities for raw, trusted, and business-ready data layers, simplifying data pipeline management.
• A Step-by-Step Guide to Mastering Modern Cloud Data Pipelines — The Medallion Architecture on Azure provides a scalable, governed framework for transforming raw data into business-ready insights.
• What is data pipeline architecture? — Data pipeline architecture is the blueprint for data flow, defined by layers, stages, and patterns tailored to use cases.
• Accelerate business insights with Lakeflow Connect, now with a Free Tier — Unifying enterprise data from disparate sources enhances AI agent capabilities by providing complete context.
• Building ETL Pipelines in Databricks | Data Engineering in Databricks — Databricks ETL pipelines transform raw data into production-ready insights using Medallion Architecture and AI-assisted coding.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Data Engineering on Medium.