Gemma 4 Local OCR Test with llama.cpp | How Accurate It Is for PDF Document Understanding (🔴 Live)

2026-04-06 · Source: Venelin Valkov · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, extended

Summary

This content explores using the Gemma 4 large language model as an Optical Character Recognition (OCR) engine, specifically for extracting information from various document types like receipts, financial reports, and academic papers. The author details the setup process, including updating the `llama.cpp` library to support Gemma 4 and configuring the `llama.cpp` server with specific image token budgets (up to 2048 tokens) and a universal batch size of 2048 for the 26B A4B model. The pipeline involves converting PDFs to images using PyPDFium, encoding them in base64, and passing them to the Gemma 4 model via an OpenAI-compatible `llama.cpp` server. Initial tests on receipts and Apple's Q1 '26 quarterly report show promising results for structured data and table extraction, even from complex layouts. However, the model struggles with precise character-level accuracy and specific numerical extraction from diagrams, occasionally producing incorrect values or empty responses on first attempts.

Key takeaway

For AI Engineers evaluating Gemma 4 for document processing, understand that while it performs well on structural extraction and complex tables, its character-level OCR accuracy can be inconsistent. You should consider integrating Gemma 4 for visual understanding and layout analysis, but for high-fidelity text extraction from digital PDFs, prioritize tools like Docling that read characters directly. For scanned or image-based documents, experiment with Gemma 4's highest image token budget, but be prepared to validate character accuracy, especially for critical numerical data.

Key insights

Gemma 4 demonstrates strong visual understanding for document structure and table extraction but struggles with character-level OCR precision.

Principles

• Pass media files first in mixed content payloads.
• Larger Gemma models support text and images; smaller models also support audio.
• LLMs excel at document structure but often miss character details.

Method

Convert PDF pages to PIL images, base64 encode them, and send to a `llama.cpp` server running Gemma 4 with optimized image token budgets for OCR tasks.

In practice

• Use PyPDFium for PDF-to-image conversion.
• Configure `llama.cpp` with `image-min-tokens`, `image-max-tokens`, and `ubatch-size`.
• Combine LLM OCR with traditional OCR for digital documents.

Topics

• Gemma 4
• Local OCR
• llama.cpp
• PDF Document Understanding
• Financial Document Extraction

Best for: AI Engineer, Machine Learning Engineer, AI Student

Related on AIssential

• Gemma 4 12B MTP Local Test | Coding, OCR, Visual RAG with llama.cpp — The Gemma 4 12B MTP model offers multimodal capabilities and efficiency, nearing larger MoE models, but with performance trade-offs.
• AI Dev 26 x SF | Jerry Liu: My Agent Can't Read a PDF? — PDFs' display-centric format poses a fundamental challenge for AI agents requiring semantic understanding of unstructured document data.
• Structured Data Retrieval with Sparrow using OCR and Vision LLM [Improved Accuracy] — Combining OCR with Vision LLMs via structured prompts improves large table data extraction accuracy and reduces hallucinations.
• Qwen3.6 Local Test | Can it Beat Gemma 4? | Coding, OCR, Image Understanding with llama.cpp | 🔴 Live — Qwen 3.6 offers strong agentic coding and visual understanding, but its verbose reasoning can impact efficiency.
• Fine-Tuning Gemma 4 for Vision — Fine-tuning Gemma 4 E2B for medical VQA improves output structure but faces challenges with complex medical reasoning.
• Running Gemma 4 Locally with Ollama on Your PC — Gemma 4 offers open-weight, locally runnable LLMs with multimodal capabilities and diverse architectures for varied hardware.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Venelin Valkov.