Building a Production RAG Ingestion Pipeline on AWS: Unstructured.io, S3 Vectors, and a Private VPC

2026-05-24 · Source: LLM on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Intermediate, medium

Summary

A production RAG ingestion pipeline built entirely on AWS addresses limitations of Bedrock Knowledge Bases, such as its 50MB file cap and poor handling of complex documents. This event-driven architecture utilizes Unstructured.io, an open-source document parsing library, running on ECS Fargate for robust layout detection, OCR, and structured chunking with metadata like page numbers. Amazon Bedrock Titan generates embeddings, and Amazon S3 Vectors stores them, offering a pay-per-use alternative to OpenSearch Serverless's \$700/month minimum cost. The entire pipeline operates within a private VPC, ensuring data never leaves AWS, requiring 8 specific VPC endpoints costing approximately \$42 per month. DynamoDB tracks document status, and SQS handles retries. The Unstructured.io Docker image, at 6.6GB, necessitates setting ECS Fargate's ephemeral storage to 50GB. An end-to-end test processed 16 chunks in 87 seconds, demonstrating effective error handling and horizontal scalability.

Key takeaway

For MLOps Engineers building production RAG systems on AWS, consider this architecture to overcome Bedrock Knowledge Bases' limitations. By self-hosting Unstructured.io on Fargate and using S3 Vectors, you gain superior document parsing, enhanced security within a private VPC, and significantly lower vector storage costs compared to OpenSearch Serverless. Ensure your ECS tasks have sufficient ephemeral storage and provision all necessary VPC endpoints, including the often-missed S3 Vectors endpoint, to avoid runtime connection issues.

Key insights

Self-hosting Unstructured.io within a private AWS VPC enables robust, secure, and cost-effective RAG ingestion.

Principles

• Prioritize structured document parsing for RAG quality.
• Design event-driven pipelines for resilience.
• Secure data by keeping all processing within a private VPC.

Method

Documents trigger SQS messages, processed by ECS Fargate running Unstructured.io, then embedded by Bedrock Titan, and stored in S3 Vectors, with DynamoDB tracking status.

In practice

• Use "urllib.parse.unquote_plus" for S3 object keys.
• Set ECS Fargate "ephemeral_storage_gib=50" for Unstructured.io.
• Ensure boto3 >= 1.39.5 for S3 Vectors client.

Topics

• RAG Ingestion Pipeline
• Unstructured.io
• AWS VPC
• S3 Vectors
• ECS Fargate
• Document Parsing
• Bedrock Titan

Code references

• SaadAlam/unstructured-rag-aws

Best for: AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• 10 Common RAG Mistakes We Keep Seeing in Production — Production RAG failures stem from ignoring document and question structure, over-relying on embeddings, and lacking generation auditability.
• Build an AI-powered website assistant with Amazon Bedrock — An AI-powered RAG system on AWS Bedrock can provide instant, role-based support from diverse data sources.
• Amazon S3 Vectors Reaches GA, Introducing "Storage-First" Architecture for RAG — S3 Vectors offers a "storage-first" approach for RAG, simplifying vector data management and scaling.
• Baseline Enterprise RAG, From PDF to Highlighted Answer — A minimal RAG pipeline can provide verifiable, sourced answers by structuring outputs and linking directly to document evidence.
• Your Documents Shouldn’t Need the Internet to Be Searchable — RAGFlow enables private, local document search and AI interaction on CPU, bypassing cloud dependencies.
• Designing a Production-Ready RAG Pipeline — Recording Now Available — Building production RAG systems requires a comprehensive architecture beyond simple prototypes.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by LLM on Medium.