Benchmarking RAG Architectures Locally on a Real Financial PDF

2026-06-20 · Source: Artificial Intelligence on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, FinTech & Digital Financial Services · Depth: Advanced, long

Summary

This article, the first in a three-part series, benchmarks retrieval-augmented generation (RAG) architectures operating entirely on local, open models without external endpoints. The project uses Akbank's 4Q2025 earnings presentation, a 38-page financial PDF with critical data embedded in charts and tables, posing significant extraction challenges. A custom pipeline employing Docling with EasyOCR and TableFormer converts the image-only PDF content into Markdown. Evaluation involves 57 questions, categorized by content type (20 text, 19 table, 18 chart), and uses RAGAS metrics, with a local qwen3:14b model as the judge. Part 1 compares three text-retrieval pipelines: Naive RAG, Hybrid RAG, and Hybrid RAG with ColBERT reranking. Results show answer correctness improving from 0.415 for Naive RAG to 0.544 for Hybrid RAG with ColBERT reranking, demonstrating the value of architectural sophistication even with local constraints.

Key takeaway

For AI Engineers building RAG systems for regulated financial documents that must run entirely in-house, you should prioritize robust data extraction and advanced retrieval architectures. Standard PDF parsing is insufficient for image-heavy content; instead, implement OCR-based pipelines like Docling with TableFormer. Furthermore, integrating Hybrid RAG with Reciprocal Rank Fusion and a ColBERT-style reranker significantly improves answer correctness, moving from 0.415 to 0.544, even with local open models. This approach is crucial for achieving reliable performance in sensitive, offline environments.

Key insights

Local RAG on complex financial PDFs requires specialized extraction and architectural choices to achieve meaningful accuracy.

Principles

• Regulated environments necessitate local-only RAG systems.
• Financial PDFs demand OCR and table reconstruction.
• Hybrid retrieval and reranking boost RAG accuracy.

Method

Convert image-only financial PDFs to Markdown using Docling, EasyOCR (full-page), and TableFormer for structured table reconstruction. Evaluate with RAGAS metrics.

In practice

• Use Docling/EasyOCR for image-heavy PDF data extraction.
• Implement Hybrid RAG with RRF for improved recall.
• Add ColBERT reranking to enhance answer correctness.

Topics

• RAG Benchmarking
• Local RAG Systems
• Financial Document AI
• OCR Data Extraction
• Hybrid Retrieval
• ColBERT Reranking

Best for: AI Engineer, Machine Learning Engineer, AI Architect

Related on AIssential

• Build 100% Local Advanced RAG System for Financial PDFs with Qwen 3.5 | Docling, LangGraph & Ollama — A local RAG system can analyze financial PDFs using open-source tools for privacy and offline capability.
• RARE: Redundancy-Aware Retrieval Evaluation Framework for High-Similarity Corpora — Existing RAG benchmarks misrepresent real-world performance due to high document redundancy and similarity in enterprise corpora.
• Article: Redesigning Banking PDF Table Extraction: A Layered Approach with Java — Reliable PDF table extraction in finance requires a layered architectural approach, not just a single parsing library.
• End-to-End Evaluation of a RAG System for Hospital Documents in Portuguese — Optimizing RAG components individually enhances performance for querying specialized documents in resource-limited settings.
• Vector RAG Is Dead. PageIndex Just Proved It. — PageIndex's tree-search framework achieves 98.7% accuracy on FinanceBench, bypassing vector RAG's limitations.
• Baseline Enterprise RAG, From PDF to Highlighted Answer — A minimal RAG pipeline can provide verifiable, sourced answers by structuring outputs and linking directly to document evidence.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence on Medium.