Uber Launches IngestionNext: Streaming-First Data Lake Cuts Latency and Compute by 25%

2026-03-25 · Source: InfoQ · Field: Technology & Digital — Data Science & Analytics, Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Intermediate, quick

Summary

Uber engineers re-architected the company's data lake ingestion platform, transitioning from a batch-oriented system to a streaming-first architecture named IngestionNext. This new platform continuously processes event streams, significantly reducing data ingestion latency from hours to minutes. The IngestionNext architecture leverages Apache Kafka for event streaming and Apache Flink jobs for processing, writing data to Apache Hudi tables with transactional capabilities. This shift enables faster data availability for analytics dashboards, experimentation platforms, and machine learning models, supporting thousands of datasets and high global data volumes. The re-architecture also improved resource efficiency, reducing compute usage by approximately 25% compared to the previous batch system.

Key takeaway

For data platform architects evaluating modernization strategies, Uber's shift to a streaming-first data lake ingestion platform demonstrates significant gains in data freshness and resource efficiency. You should consider adopting similar streaming architectures, leveraging technologies like Apache Kafka, Flink, and Hudi, to accelerate data availability for critical analytics and machine learning applications, while also optimizing compute costs.

Key insights

Streaming data ingestion reduces latency and improves data freshness for analytics and ML workloads.

Principles

• Data freshness is a key dimension of data quality.
• Continuous processing can optimize resource utilization.

Method

Implement a streaming pipeline using Apache Kafka and Flink, writing to Hudi tables with transactional commits, and managing file compaction for efficiency.

In practice

• Use Apache Hudi for transactional data lake operations.
• Implement compaction for small file issues in streaming.
• Track offsets for reliable recovery in streaming jobs.

Topics

• Data Lake Ingestion
• Streaming Data
• Apache Flink
• Apache Hudi
• Data Freshness

Code references

• apache/hudi

Best for: VP of Engineering/Data, AI Engineer, Machine Learning Engineer, Data Engineer, MLOps Engineer, AI Architect

Related on AIssential

• How Uber Built Big Data System — From a Few TBs to 350 Petabytes with Sub-Hour Latency — Uber's data platform evolution highlights re-architecting for mutable data at petabyte scale, leading to Apache Hudi's creation.
• Pinterest’s CDC-Powered Ingestion Slashes Database Latency from 24 Hours to 15 Minutes — CDC-powered ingestion dramatically reduces data latency and infrastructure costs by processing only changed records.
• Scaling for MHHS: how Octopus Energy achieved a 50x cost reduction in margin data engineering — Re-architecting data pipelines for granular processing and incremental updates drastically reduces costs and improves data freshness.
• Ingesting Data into Databricks | Data Engineering in Databricks — Databricks supports both direct file uploads and external cloud storage connections for data ingestion into Delta tables.
• Batch or Stream? The Eternal Data Processing Dilemma — Data processing choice hinges on data freshness value and the speed required for action.
• Apache Kafka for Data Engineers — Beginner Guide — Apache Kafka provides a scalable, fault-tolerant, and decoupled architecture for real-time data streaming.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by InfoQ.