Is this the easiest way to use VLMs for RAG? #llm #rag #retrievalaugmentedgeneration

2025-10-24 · Source: Nicholas Renotte · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, short

Summary

The article demonstrates using Vision Language Models (VLMs) with the Dockling tool to extract complex information from PDF documents for Retrieval Augmented Generation (RAG) pipelines. It highlights the challenges of processing documents containing mixed content like text, images, tables, formulas, and code. The author uses a stock announcement PDF for an ETF as an example, showcasing how Dockling, specifically with the `VLM-OD` and `granite dockling` models, can convert such a document into a structured Markdown format. The process, executed via a UV run command in VS Code, successfully extracted a complex table and specific data points, like an interest percentage of 10.80.183%, in approximately 20.68 seconds, demonstrating efficient and accurate content parsing.

Key takeaway

For AI Engineers building RAG systems that process diverse document types, integrating tools like Dockling with VLMs can significantly streamline data extraction. Your team can automate the conversion of complex PDFs, including those with tables and images, into structured formats like Markdown, reducing manual preprocessing effort and improving the quality of retrieval. Consider experimenting with Dockling's VLM pipeline to enhance your document ingestion workflow.

Key insights

VLMs simplify complex document parsing for RAG by efficiently extracting mixed content into structured formats.

Principles

• VLMs excel at multimodal document understanding.
• Structured output improves RAG pipeline efficiency.

Method

Use Dockling with a VLM pipeline (e.g., `VLM-OD` and `granite dockling`) to convert complex PDFs into Markdown, preserving tables and images for RAG.

In practice

• Convert PDFs to Markdown with Dockling.
• Extract tables and images automatically.
• Integrate VLM output into RAG pipelines.

Topics

• Vision Language Models
• Retrieval-Augmented Generation
• Document Processing
• Dockling
• Markdown Conversion

Best for: Machine Learning Engineer, AI Engineer, Data Scientist

Related on AIssential

• Convert Any Document To LLM Knowledge with Docling & Ollama (100% Local) | PDF to Markdown Pipeline — A local pipeline converts complex PDFs to markdown with tables and image descriptions for RAG applications.
• 7 Steps to Mastering Retrieval-Augmented Generation — Mastering RAG involves a seven-step process from data preparation to grounded answer generation.
• Vision LLMs are PDF Parsers Too: Reading Charts and Diagrams for RAG — Vision LLMs make image content searchable for RAG, complementing text parsers despite trade-offs in cost and exactness.
• How NTT DATA Transforms Enterprise Document Parsing with LlamaIndex — LlamaIndex streamlines document parsing and RAG solutions, significantly boosting efficiency and accuracy in enterprise AI development.
• Making a PDF’s Images Searchable for RAG, Without Paying to Read Them All — Cost-ordered image analysis for RAG prioritizes cheap filters and OCR before costly vision models.
• Your RAG Pipeline Is Probably Useless. Here’s a Better Alternative — RAG pipelines fail predictably; alternatives like long-context, summarization, structured, or graph-based retrieval offer better solutions.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Nicholas Renotte.