How LinkedIn Rebuilt Search with AI: New AI Based Architecture

2026-07-12 · Source: Machine Learning on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Data Science & Analytics · Depth: Intermediate, extended

Summary

LinkedIn has fundamentally rebuilt its search architecture, moving from keyword matching to a semantic understanding of user intent to handle millions of real-time queries per second. The new AI-based system comprises five stages: Query Understanding, Embedding-Based Retrieval (EBR), Small Language Model (SLM) Ranking, an Auction Layer, and Explainability. It leverages a bi-encoder for fast, broad retrieval of ~1,000 candidates and a 0.6B parameter cross-encoder SLM for precise ranking, distilled from larger 7B and 1.7B teacher models. Offline and nearline pipelines using Spark, Flyte, and Flink precompute embeddings and summaries. Cost optimizations like model pruning, summarization, and embedding compression achieved a ~75x throughput increase with minimal quality impact, making the system economically viable. A continuous LLM Judge system ensures quality monitoring and training data generation.

Key takeaway

For AI Engineers designing large-scale semantic search or recommendation systems, LinkedIn's architecture demonstrates that separating fast, broad retrieval from precise ranking is critical. You should prioritize model distillation (e.g., 7B to 0.6B parameters) and aggressive cost optimizations like embedding compression to achieve high throughput (e.g., 22,000 items/sec/GPU) with minimal quality degradation. Implement an LLM judge early to establish robust quality metrics (NDCG@10, AUC) and generate continuous training data, ensuring your system evolves effectively.

Key insights

LinkedIn's AI search architecture shifts from keyword matching to semantic intent understanding via a multi-stage funnel for scale and precision.

Principles

• Separate fast, broad retrieval (recall) from slow, precise ranking (precision).
• Distill knowledge from large teacher models into smaller, production-viable student models.
• Precompute embeddings, summaries, and indexes offline or nearline to minimize online latency.

Method

The system processes queries through an LLM-powered understanding layer, then uses a bi-encoder for embedding-based retrieval (EBR) to find ~1,000 candidates, followed by a 0.6B parameter SLM cross-encoder for precise ranking, and finally an auction layer.

In practice

• Implement a dual-tower (bi-encoder) model for high-throughput embedding-based retrieval.
• Employ a cross-encoder SLM for deep, accurate ranking on a reduced candidate set.
• Utilize an LLM judge system to generate relevance labels and monitor search quality continuously.

Topics

• Semantic Search
• AI Architecture
• Large Language Models
• Embedding-Based Retrieval
• Model Distillation
• Search Ranking

Best for: NLP Engineer, CTO, VP of Engineering/Data, AI Engineer, Machine Learning Engineer, AI Architect

Related on AIssential

• Rethinking Negative Sampling for Knowledge Distillation in Retrieval, A Distillation Recipe for Small Language Model Search Agents, and More! — Effective information retrieval advancements often stem from rethinking fundamental assumptions and optimizing data representation or model behavior.
• How a simple AI search architecture I made defined industry standard — A custom AI search architecture standardized influencer categorization using semantic search and a novel confidence scoring formula.
• Generative Engine Optimization: A Technical Framework for AI Search Visibility — Digital visibility now hinges on content being selected as a trusted source by generative AI systems.
• From TF-IDF to Transformers: Implementing Four Generations of Semantic Search — Semantic search evolved from explicit rules to learned contextual representations, balancing interpretability with semantic depth.
• Embedding Model Selection: 10 Scenario-Based Questions & Solutions — The most reliable embedding model selection combines public benchmark filtering with specific business data experimentation.
• Search Ranking with Machine Learning: Learning to Rank - A Complete Introduction — Machine Learning transforms search ranking from static rule-based systems into dynamic, context-aware, and personalized experiences.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning on Medium.