Indexing the Data Lake for Online Point Queries

2026-07-27 · Source: Spotify Engineering · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure, Software Development & Engineering · Depth: Advanced, long

Summary

Random Access Parquet (RAP) addresses the challenge of performing fast point queries on vast data lakes, a critical need for online services and AI agents at companies like Spotify. While cloud storage like GCS and S3 Express offers single-digit millisecond latency, traditional distributed SQL engines introduce seconds of overhead for single-row lookups. RAP bridges this gap by employing an external index that maps keys directly to file locations within existing Parquet files, enabling precise ranged reads. This allows petabytes of data in GCS data lakes to be queried interactively, avoiding the economic burden of moving exabytes into expensive key-value stores like Bigtable, and supporting use cases from user listening history to AI agent context retrieval.

Key takeaway

For MLOps Engineers or AI Architects building online services that require low-latency access to extensive historical data in cloud data lakes, you should evaluate Random Access Parquet (RAP). This approach allows you to perform fast point queries on exabyte-scale datasets without the prohibitive cost and complexity of duplicating data into traditional key-value stores. Implementing RAP can significantly expand the context window for AI agents and enable new interactive features directly from your data lake.

Key insights

Random Access Parquet enables fast point queries on data lakes by using an external index to eliminate scan bottlenecks.

Principles

• Collapsing dependent reads saves latency and bandwidth.
• External indexes provide definitive key-to-location mappings.
• Optimizing Parquet files reduces read operations.

Method

RAP builds an external index mapping keys to file and row numbers. It then uses cached file metadata and precise ranged reads to fetch data, issuing parallel reads to avoid dependent load chains.

In practice

• Sort Parquet files by key for data concentration.
• Store point query fields as single blobs (JSON, Protobuf).
• Hoist small values into the index for zero-read queries.

Topics

• Random Access Parquet
• Data Lake
• Point Queries
• Parquet File Format
• External Indexing
• Cloud Storage
• AI Agents

Code references

• apache/parquet-format

Best for: CTO, VP of Engineering/Data, Director of AI/ML, Data Engineer, MLOps Engineer, AI Architect

Related on AIssential

• Building Your Modern Data Analytics Stack with Python, Parquet, and DuckDB — Combining Parquet for columnar storage and DuckDB for direct querying offers a fast, efficient data analytics stack.
• Databricks Lakebase LTAP: The Data Architecture AI Applications Actually Need — LTAP unifies transactional and analytical data at the storage layer, enabling different engines to operate on a single source of truth.
• Hydrolix brings high-speed analytics to petabyte-scale agentic AI — Hydrolix provides sub-second query latency across petabyte-scale data for agentic AI, enabling real-time natural language insights.
• From data lake to AI-ready analytics: Introducing new data source with S3 Tables in Amazon Quick — Amazon Quick now directly queries Apache Iceberg tables in S3, enabling near real-time, AI-powered analytics on data lakes.
• How CockroachDB Built Vector Indexing at Scale — Integrating vector indexes as native table data within a distributed SQL database simplifies scaling and maintenance.
• Query Tags: The Context Your Warehouse Queries Have Been Missing — Query Tags provide granular business context to SQL executions, enhancing workload traceability and cost attribution.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Spotify Engineering.