Building a Production RAG Pipeline: Webhooks, Deduplication, and 40k Documents

2026-05-30 · Source: Data Engineering on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Cloud Computing & IT Infrastructure · Depth: Advanced, medium

Summary

Building a production RAG pipeline for an enterprise workplace safety training platform, scaled to forty thousand documents, involves significant data engineering challenges. The architecture features two distinct ingestion paths: an asynchronous, event-driven file upload via Azure Event Grid for bulk content migration, and a synchronous HTTPS webhook for real-time content updates from a CMS. Both paths converge on an ingestion core that performs Azure Content Safety checks, content parsing, chunking, and atomic deduplication. The system employs a dual embedding strategy, using 1536-dimensional vectors for permanent storage and 128-dimensional vectors for faster query-time retrieval, all persisted in PostgreSQL with the pgvector extension and HNSW indexes. A critical silent deduplication bug, caused by duplicate event subscriptions and non-atomic upserts, degraded retrieval quality by creating redundant embeddings.

Key takeaway

For MLOps Engineers building production RAG systems with continuously updating knowledge bases, prioritize robust data engineering. Model each data flow independently, enforce atomicity at every read-write boundary to prevent silent data corruption, and proactively instrument your vector store with metrics like chunk counts. This approach helps detect subtle issues like duplicate embeddings early, ensuring retrieval quality and system reliability before degradation becomes noticeable.

Key insights

Production RAG pipelines are primarily data engineering challenges, requiring robust solutions for continuous updates and data integrity.

Principles

• Production RAG pipelines require distinct ingestion speeds for bulk and incremental updates.
• Content safety checks should be applied at both ingestion and query layers.
• Dual embedding strategies can balance retrieval quality and query latency.

Method

The pipeline uses Azure Event Grid for async file uploads and direct HTTPS webhooks for CMS updates, converging on a core that performs Azure Content Safety scanning, parsing, chunking, and atomic upsert deduplication before generating dual embeddings (1536D for storage, 128D for query) in PostgreSQL with pgvector and HNSW indexes.

In practice

• Implement atomic upsert operations for deduplication in vector stores.
• Audit event subscriptions to prevent duplicate triggers in event-driven systems.
• Instrument vector stores with chunk count metrics to detect silent data issues.

Topics

• RAG Pipeline Architecture
• Data Ingestion
• Deduplication
• Vector Databases
• Azure Event Grid
• Content Safety

Best for: AI Engineer, MLOps Engineer, Data Engineer

Related on AIssential

• The Hidden Engineering Nightmares of Managing Embeddings in Production RAG Pipelines (and How to… — Disciplined embedding infrastructure, not just the latest models, is critical for reliable and scalable production RAG systems.
• RAG in Production: Designing Retrieval Pipelines That Stay Accurate as Your Data Changes — Production RAG systems require dynamic, continuous retrieval pipelines to maintain accuracy with evolving data.
• Building a Production ready RAG API — Production RAG APIs require robust backend engineering to handle ingestion, streaming, rate limits, memory, and caching at scale.
• We Built a Complete RAG Evaluation System With Zero Paid APIs — Dataset Creation, Metrics, and… — A complete, local RAG evaluation system provides dataset creation, human review, and four metrics without external APIs.
• How to Build an Elastic Vector Database with Consistent Hashing, Sharding, and Live Ring Visualization for RAG Systems — Consistent hashing with virtual nodes enables scalable vector storage with minimal data movement during topology changes.
• The Data Blueprint for 2026: Architecting High-Fidelity Pipelines for Generative AI — Generative AI demands a new data architecture focused on real-time, high-fidelity data, human validation, and hardware-aware optimization.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Data Engineering on Medium.