How We Reduced a Large Spark ETL Job Runtime by 80%: A Practical Databricks Optimization Guide

2026-03-11 · Source: Data Engineering on Medium · Field: Technology & Digital — Data Science & Analytics, Software Development & Engineering, Artificial Intelligence & Machine Learning · Depth: Intermediate, medium

Summary

A Databricks optimization guide details how a large Spark ETL pipeline's runtime was reduced by approximately 80%, from hours to minutes, without altering business logic or cluster size. The optimization journey addressed a critical ETL step that degraded significantly after a data growth event doubled row counts. Key improvements included removing calculations from join conditions, simplifying COUNT(DISTINCT) aggregations, and understanding shuffle spill and data skew. The article also covers the importance of up-to-date table statistics, Z-ordering, and increasing shuffle partitions. The most impactful technique was salting skewed keys, which alone reduced runtime by ~40% and eliminated disk/memory spill. Overall, these practical debugging techniques, leveraging Spark UI and execution plans, led to substantial cost savings and increased reliability.

Key takeaway

For Data Engineers managing large-scale Spark ETL pipelines on Databricks, systematically analyzing Spark UI for bottlenecks like join conditions, complex aggregations, and data skew is crucial. Prioritize pre-computing join keys and simplifying aggregations. If spill and skew persist, consider increasing `spark.sql.shuffle.partitions` and implementing advanced techniques like salting to distribute skewed keys, which can yield significant runtime reductions and cost savings.

Key insights

Systematic Spark ETL optimization can drastically reduce runtime and costs without changing business logic or cluster size.

Principles

• Avoid calculations in join conditions.
• Simplify aggregations before complex operations.
• Trust Spark's optimizer unless evidence dictates otherwise.

Method

Diagnose Spark ETL bottlenecks using Spark UI, execution plans, and SQL improvements. Address issues like complex joins, inefficient aggregations, shuffle spill, and data skew, potentially using salting for skewed workloads.

In practice

• Pre-compute join keys upstream of joins.
• Pre-label rows for complex COUNT(DISTINCT).
• Use salting for highly skewed data keys.

Topics

• Spark ETL
• Databricks Optimization
• Data Skew
• Shuffle Spill
• Join Optimization

Best for: Data Engineer, MLOps Engineer, Machine Learning Engineer

Related on AIssential

• Predictive Optimization at Scale: A Year of Innovation and What’s Next — Predictive Optimization automates data layout and maintenance for cost-effective, high-performance lakehouses.
• Debunking 8 data layout myths: why Liquid Clustering outperforms partitioning — Liquid Clustering provides a flexible, performant, and scalable data layout for modern Lakehouses, overcoming partitioning's limitations.
• Full End-to-End Data Engineering Project in Databricks — Automating data ingestion and ETL pipelines in Databricks creates a robust, self-updating data engineering workflow.
• Apache Spark Mastery (Part 3of 6): Wrangling Messy Data — Effective data wrangling in Spark involves systematic cleaning, standardization, and handling of diverse data structures.
• Scaling ML Inference on Databricks: Liquid or Partitioned? Salted or Not? — Data layout, specifically file partitioning and row limits, critically impacts ML inference scalability on Databricks.
• 10 Data Ingestion Optimization Techniques for Faster and Scalable Processing — Parallelizing data ingestion workloads is critical for achieving faster, more scalable processing and reducing latency by over 50%.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Data Engineering on Medium.