How MakeMyTrip Achieved Millisecond Personalization at Scale with Databricks

2026-04-07 · Source: Databricks · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Cloud Computing & IT Infrastructure · Depth: Intermediate, short

Summary

MakeMyTrip, India's largest online travel agency, has implemented Databricks' Real-Time Mode (RTM) in Apache Spark Structured Streaming to achieve millisecond-level latency for its "last-searched" hotels feature. This feature provides real-time, personalized recommendations to millions of daily users across consumer and corporate travel. Previously, Apache Spark's micro-batch mode delivered latencies of one to two seconds, which was too slow. While Apache Flink met latency requirements, MakeMyTrip avoided a dual-engine architecture due to concerns about architectural fragmentation, duplicated business logic, higher operational overhead, consistency risks, and increased infrastructure costs. RTM's innovations, including continuous data flow, pipeline scheduling, and streaming shuffle, enable Spark to process data as it arrives, eliminating micro-batch latency bottlenecks and allowing MakeMyTrip to maintain a single, cost-effective Spark-based pipeline.

Key takeaway

For AI Architects and Data Engineers designing real-time personalization systems, consider adopting Apache Spark's Real-Time Mode (RTM) to achieve millisecond-level latencies. This approach allows you to avoid the architectural complexity and operational overhead of a dual-engine setup (e.g., Spark + Flink) while maintaining a unified, cost-effective data pipeline. Evaluate RTM for use cases requiring continuous data flow and immediate processing to enhance user experience and reduce friction.

Key insights

Achieving millisecond-level real-time personalization at scale is possible with Apache Spark's Real-Time Mode.

Principles

• Avoid architectural fragmentation.
• Unified data processing reduces complexity.
• Continuous data flow minimizes latency.

Method

MakeMyTrip's architecture merges B2C/B2B clickstream topics, applies personalization logic via RTM, performs low-latency stateful lookups in Aerospike, and pushes results to Redis for sub-50ms serving.

In practice

• Use RTM for sub-second Spark streaming.
• Consolidate batch and real-time engines.
• Employ Aerospike for stateful lookups.

Topics

• MakeMyTrip
• Real-Time Personalization
• Databricks Real-Time Mode
• Apache Spark Structured Streaming
• Low-Latency Data Processing

Best for: Data Engineer, AI Engineer, AI Architect

Related on AIssential

• Announcing General Availability of Real-Time Mode for Apache Spark Structured Streaming on Databricks — Spark Structured Streaming's Real-Time Mode delivers millisecond latency, unifying batch and streaming workloads.
• Apache Spark Real-Time Mode for Gaming: A Better Way to Do Real-Time Sessionization — Apache Spark Real-Time Mode with "transformWithState" unifies reactive and proactive stateful stream processing for sub-second latency.
• Build hyper-personalized software for an audience of one — A Raspberry Pi and LLM enable a personalized, typo-tolerant, and unobtrusive to-do system.
• 7 Minutes to Understand the New Spark Streaming Feature that Changes Everything — Spark Structured Streaming's new Real-Time Mode achieves millisecond latency by optimizing data consumption, stage scheduling, and data shuffling.
• Building Real-Time AI Agent with Apache Flink — Integrating LLMs directly into streaming pipelines with Apache Flink enables real-time, intelligent agent decision-making.
• 30+ Updates per Second per Account: Uber Scales Ledger Processing with Batching — Batching financial ledger updates with Redis coordination enables high-throughput processing while maintaining strict consistency.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Databricks.