Scaling for MHHS: how Octopus Energy achieved a 50x cost reduction in margin data engineering

2026-05-23 · Source: Databricks · Field: Technology & Digital — Data Science & Analytics, Cloud Computing & IT Infrastructure, Software Development & Engineering · Depth: Intermediate, medium

Summary

Octopus Energy re-architected its margin data pipelines to address the UK's Market-wide Half-Hourly Settlement (MHHS) regulation, which increased data points by 48x for its over 8 million customers. Facing a projected \$1 million annual cost increase and a \$23.63 cost per settlement date under its legacy system, the company developed a new architecture. This involved three specialized data streams for half-hourly settlement, smart tariff customers, and monthly standard tariff customers, all underpinned by a unified consumption layer. A key optimization was implementing Delta Lake's Change Data Feed, reducing processed rows from 25 billion to 300 million (a 98.8% reduction). Combined with targeted Spark and Delta optimizations and the use of Databricks Serverless, Octopus Energy achieved a 50x cost reduction, bringing the cost per settlement date down to \$0.48, which is 2x cheaper than the original legacy system, while improving data freshness from weekly to daily. The project was completed in three months.

Key takeaway

For Directors of AI/ML or Data Engineers scaling data infrastructure due to increased granularity, you must re-evaluate your architecture beyond simply adding compute. Aligning processing to natural data grains and implementing incremental processing, like Delta Lake's Change Data Feed, can yield massive cost reductions and improve data freshness. Audit existing optimizations and trust built-in query optimizers before adding custom logic to avoid unnecessary overhead.

Key insights

Re-architecting data pipelines for granular processing and incremental updates drastically reduces costs and improves data freshness.

Principles

• Align processing to natural data grains.
• Incremental processing transforms pipeline economics.
• Audit and remove unjustified compute operations.

Method

Octopus Energy re-architected into three specialized streams (Settlement, Half-Hourly, Monthly) with a unified consumption layer, using Delta Lake's Change Data Feed for incremental processing and targeted Spark/Delta optimizations.

In practice

• Implement Change Data Feed for 98.8% row reduction.
• Use broadcast joins for reference tables <500MB.
• Enable Liquid clustering for filter/join columns.

Topics

• Market-wide Half-Hourly Settlement
• Data Engineering
• Delta Lake Change Data Feed
• Apache Spark Optimization
• Databricks Serverless
• Energy Transition
• Cost Reduction

Best for: Data Engineer, Director of AI/ML, Consultant

Related on AIssential

• The Evolution of Data Engineering: How Serverless Compute is Transforming Notebooks, Lakeflow Jobs, and Spark Declarative Pipelines — Databricks serverless compute automates infrastructure management, offering significant cost savings and performance improvements.
• The Undo Button for Big Data: An Intro to Delta Lake Time Travel — Delta Lake's transaction log enables data time travel, providing an undo button for big data operations.
• Uber Launches IngestionNext: Streaming-First Data Lake Cuts Latency and Compute by 25% — Streaming data ingestion reduces latency and improves data freshness for analytics and ML workloads.
• The Memory Crunch Is Stress-Testing Enterprise AI — Enterprise AI's true challenge lies in fragmented, inconsistent data foundations, not model quality.
• Debunking 8 data layout myths: why Liquid Clustering outperforms partitioning — Liquid Clustering provides a flexible, performant, and scalable data layout for modern Lakehouses, overcoming partitioning's limitations.
• How Figma Upgraded Data Pipeline from Multi-Day Latency to Real-Time — Incremental synchronization via Change Data Capture significantly improves data freshness and reduces costs for large-scale data pipelines.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Databricks.