How Snapchat Serves a Billion Predictions Per Second

2025-12-15 · Source: ByteByteGo Newsletter · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Data Science & Analytics · Depth: Advanced, long

Summary

Snapchat's machine learning platform, Bento, serves over a billion predictions per second for 477 million daily users, handling critical decisions for content feeds, ad auctions, friend suggestions, and AR lenses within 100 milliseconds. The platform is engineered to manage immense latency, scale, freshness, and iteration pressures. Bento's training half utilizes Kubeflow, a layered code structure for hundreds of daily experiments, and a unique model export that splits compute graphs for GPU/CPU optimization. The serving half, powered by Robusta (Apache Spark), processes 10 trillion events/day, serving 1 TB/second of reads from 800 TB of online feature data. It employs strategies like feature collocation or a dedicated Retrieval service for high fanout, alongside inference optimizations such as compute graph splitting and data plane optimization, which achieved 2x lower latency and 10x cheaper data plane costs. A continuous feedback loop with robust monitoring and a Kubernetes-like deployment control plane ensures system stability and continuous model improvement.

Key takeaway

For MLOps Engineers scaling high-volume recommendation or ranking systems, you must architect for asymmetric workloads where one request expands into many model evaluations. Prioritize robust feature store synchronization to prevent train/serve skew, a common operational pitfall. Consider advanced optimizations like splitting compute graphs for GPU/CPU and redesigning inference APIs for raw byte feature transfer, which can yield 2x lower latency and 10x cheaper data plane costs, ensuring your platform can absorb significant growth.

Key insights

Snapchat's Bento platform scales ML ranking to a billion predictions/second by optimizing for asymmetric workloads and continuous iteration.

Principles

• Asymmetric ranking workloads demand specialized architectural solutions.
• Feature store consistency prevents train/serve skew.
• Data plane optimization can significantly reduce latency and cost.

Method

Bento's training pipeline uses Kubeflow to orchestrate data generation, model training (GPU/TPU), evaluation, and a split compute graph export.

In practice

• Collocate document features on inference instances for low latency.
• Split compute graphs for GPU/CPU to optimize resource use.
• Redesign inference APIs for raw byte feature transfer.

Topics

• Machine Learning Platforms
• Recommendation Systems
• Feature Stores
• Real-time Inference
• MLOps
• Data Plane Optimization

Best for: AI Architect, CTO, VP of Engineering/Data, AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Optimizing ML Workload Network Efficiency (Part I): Feature Trimmer — Sending only necessary features, guided by model signatures, drastically cuts network overhead in ML serving.
• Versioned Late Materialization for Ultra-Long Sequence Training in Recommendation Systems at Scale — Versioned late materialization eliminates UIH data redundancy by reconstructing sequences just-in-time from a single immutable source using lightweight versioned pointers.
• The TechBeat: Optimizing a Fast Feature Store for Costs: ShareChat's Lessons Learned (6/21/2026) — ShareChat achieved 10x cost reduction in its 1B features/sec ML feature store through ScyllaDB, cloud tuning, protobuf, and profiling.
• Your Data Pipeline Works Until It Suddenly Doesn’t — Slow data pipelines are effectively broken pipelines for real-time AI and analytics.
• Slack Outlines Four-Phase Journey to a Multi-Cloud AI Serving Platform — Multi-cloud AI serving platforms enhance resilience, performance, and model access by abstracting provider dependencies.
• Scaling Recommendation Systems with Request-Level Deduplication — Request-level deduplication is a cross-cutting technique that optimizes storage, training, and serving for large-scale recommendation systems.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by ByteByteGo Newsletter.