From Flutter On-Device to Gemini Vision: Building a Multi-Language Document Extraction Engine

2026-06-27 · Source: LLM on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Data Science & Analytics · Depth: Intermediate, long

Summary

A solo developer achieved 85-95% transaction extraction accuracy and 92% category accuracy for a multi-language document extraction engine, supporting 5 languages and over 20 document formats. Initially, the project struggled with 40% accuracy using Flutter on-device text extraction, which failed on image-based PDFs, and Google Document AI, which distorted table structures and over-fit to English formats. The breakthrough involved a hybrid Vision LLM approach, primarily using Gemini 2.5 Flash with Claude Sonnet as a fallback. This method combines PDF images for structural understanding and raw OCR text/numeric tokens for precise financial data, significantly reducing manual review to 10-15% and eliminating code changes for new document formats. Aggressive caching and targeted repair calls further optimized costs and reliability.

Key takeaway

For AI Engineers building document extraction systems, prioritize a hybrid Vision LLM and OCR approach from the outset. You should combine Vision LLMs for structural understanding and OCR for numeric precision to achieve high accuracy across diverse document formats and languages. This strategy minimizes development time for new formats and significantly reduces manual review, avoiding the pitfalls of text-only parsing or on-device solutions.

Key insights

Hybrid Vision LLM and OCR excels at structured data extraction from diverse, unstructured documents by combining visual and numeric accuracy.

Principles

• Document structure is runtime-discovered, not assumed by file type.
• Combine Vision LLM for structure with OCR for numeric accuracy.
• Aggressive caching significantly reduces LLM API costs.

Method

Send PDF images for structure, raw OCR text for context, and parsed numeric tokens for anchoring to a Vision LLM. Implement a multi-stage fallback chain with repair calls and aggressive caching.

In practice

• Test Vision LLMs on actual data before assuming limitations.
• Design multi-currency reconciliation as a first-class constraint.
• Implement aggressive caching (40-60% hit rate) for LLM calls.

Topics

• Document Extraction
• Vision LLMs
• Hybrid AI Architectures
• Financial Data Processing
• LLM Cost Optimization
• Multi-language Systems

Best for: AI Engineer, Machine Learning Engineer, Director of AI/ML

Related on AIssential

• OCR vs LLM Extraction — The Evolution of Document AI — OCR digitizes text, while LLMs understand document meaning and extract contextual information.
• How Grab Built a Vision LLM to Scan Images — Custom-built, lightweight Vision LLMs can surpass larger general-purpose models for specialized document processing tasks.
• LlamaParse vs  LLMs: Live OCR Battleground — Complex document parsing requires a hybrid approach combining traditional OCR with VLMs and structured harnesses to overcome LLM limitations.
• Extracting BOMs from Electrical Drawings with AI: Azure OpenAI GPT-5.4 + Azure Document Intelligence — Decomposing complex visual tasks into smaller, hybrid AI stages significantly improves accuracy and efficiency.
• DeepSeek OCR Review — DeepSeek OCR offers rapid, accurate document-to-markdown conversion, outperforming vision LLMs in speed and data fidelity.
• The PDF Paradox: Why Document Parsing Is Still Hard — And Why the Hybrid Stack Is Winning — PDF's fixed-layout design, while excellent for rendering, creates a significant parsing challenge for AI.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by LLM on Medium.