Scaling Recommendation Systems with Request-Level Deduplication

2026-04-13 · Source: Pinterest Engineering Blog - Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Cloud Computing & IT Infrastructure · Depth: Advanced, medium

Summary

Pinterest implemented request-level deduplication to manage the infrastructure demands of scaling its recommendation models, such as the Foundation Model, which saw a 100x increase in transformer dense parameter counts and a 10x increase in model dimension. This family of techniques ensures request-level data is processed and stored once, significantly reducing costs across the ML lifecycle. For storage, leveraging Apache Iceberg with user ID and request ID sorting achieved 10-50x compression on user-heavy feature columns. In training, solutions like Synchronized Batch Normalization (SyncBatchNorm) and user-level masking addressed issues arising from non-IID data, leading to a 4x end-to-end training speedup for retrieval and a ~2.8x speedup for ranking. For ranking serving, the Deduplicated Cross-Attention Transformer (DCAT) architecture, implemented with custom Triton kernels, delivered a 7x increase in throughput.

Key takeaway

For MLOps Engineers or AI Architects scaling recommendation systems, you should prioritize implementing request-level deduplication. This approach significantly reduces storage, training, and serving costs, enabling larger models without proportional infrastructure increases. Consider adopting techniques like Apache Iceberg for data storage, SyncBatchNorm and user-level masking for training, and the DCAT architecture for ranking models to achieve substantial throughput gains and cost efficiencies.

Key insights

Request-level deduplication is a cross-cutting technique that optimizes storage, training, and serving for large-scale recommendation systems.

Principles

• Request-level deduplication is a cross-cutting technique.
• Simple fixes can unlock significant performance gains.
• Impact compounds across the ML stack.

Method

Implement request-level deduplication by sorting data with Apache Iceberg, applying SyncBatchNorm and user-level masking for training, and using a two-tower architecture for retrieval and DCAT for ranking/serving.

In practice

• Use Apache Iceberg for 10-50x storage compression.
• Apply SyncBatchNorm to restore IID statistics.
• Implement user-level masking for InfoNCE loss.

Topics

• Recommendation Systems
• Data Deduplication
• ML Infrastructure
• Apache Iceberg
• Batch Normalization
• Transformer Architectures
• MLOps

Best for: Machine Learning Engineer, MLOps Engineer, AI Architect

Related on AIssential

• How Snapchat Serves a Billion Predictions Per Second — Snapchat's Bento platform scales ML ranking to a billion predictions/second by optimizing for asymmetric workloads and continuous iteration.
• Finding Duplicates in Tabular Data with Jupyter and Prodigy — Effective data deduplication for "near duplicates" requires a human-in-the-loop, supported by tools that reduce candidate pairs and customize the labeling experience.
• Building a Production RAG Pipeline: Webhooks, Deduplication, and 40k Documents — Production RAG pipelines are primarily data engineering challenges, requiring robust solutions for continuous updates and data integrity.
• Versioned Late Materialization for Ultra-Long Sequence Training in Recommendation Systems at Scale — Versioned late materialization eliminates UIH data redundancy by reconstructing sequences just-in-time from a single immutable source using lightweight versioned pointers.
• Making User-Sequence Data More Cost-Efficient, Faster, and Easier to Use — A unified definition for user sequences ensures consistency across real-time indexing, batch processing, and online serving.
• MLT-Dedup: Efficient Large-Scale Online Video Deduplication via Multi-Level Representations and Spatial-Temporal Matching — MLT-Dedup efficiently identifies near-duplicate videos using multi-level representations and spatial-temporal matching for improved online platform performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Pinterest Engineering Blog - Medium.