Comparative Study of Domain-adapted VLMs for General Document Visual Question Answering

2026-07-08 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

This study evaluates 8 open-source pretrained Vision-Language Models (VLMs) on Document Visual Question Answering (DocVQA) across three distinct document domains: industrial documents, infographics, and presentation slides. The research systematically assesses model performance through zero-shot evaluations, fully supervised finetuning (including inter- and intra-dataset evaluations), and few-shot learning experiments to gauge knowledge transfer. Findings indicate that large pretrained VLMs exhibit strong zero-shot baselines for structured layouts but show significant performance drops on visually complex layouts like infographics and slides. While parameter scaling is a dominant factor, smaller architectures achieve higher relative gains from supervised finetuning. Crucially, cross-domain and few-shot experiments reveal that visual understanding, rather than VLM knowledge, is the primary bottleneck for DocVQA. Models finetuned with just 50 target domain samples rapidly adapt, occasionally surpassing fully supervised counterparts.

Key takeaway

For Machine Learning Engineers developing Document Visual Question Answering (DocVQA) solutions, recognize that visual understanding is your primary bottleneck, not VLM knowledge. You should prioritize finetuning strategies, as even 50 target domain samples can enable rapid adaptation, sometimes outperforming fully supervised models. Consider smaller VLM architectures, which show higher relative performance gains from supervised finetuning, to optimize resource utilization and deployment on complex document layouts like infographics and slides.

Key insights

Visual understanding is the main bottleneck for DocVQA, not VLM knowledge, with few-shot adaptation proving highly effective.

Principles

• VLM performance decreases on visually complex layouts.
• Supervised finetuning yields higher gains in smaller VLM architectures.
• Visual understanding is DocVQA's primary bottleneck.

Method

The study systematically assesses 8 VLMs using zero-shot, fully supervised finetuning (inter- and intra-dataset), and few-shot learning evaluations for knowledge transfer across document domains.

In practice

• Finetune VLMs with 50 target domain samples for rapid adaptation.
• Prioritize visual understanding improvements for DocVQA tasks.
• Consider smaller VLMs for higher relative finetuning gains.

Topics

• Document Visual Question Answering
• Vision-Language Models
• Few-shot Learning
• Zero-shot Evaluation
• Supervised Finetuning
• Multimodal AI

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Noise-Aware Visual Representation Learning for Medical Visual Question Answering — Learning robust visual representations through denoising enhances Medical Visual Question Answering performance and robustness.
• Dr. DocBench: A Comprehensive Benchmark for Expert-Level and Difficult Document Parsing — Current document parsing benchmarks fail to assess expert-level, complex documents, revealing a critical gap in VLM capabilities.
• From Codebooks to VLMs: Evaluating Automated Visual Discourse Analysis for Climate Change on Social Media — VLMs can effectively analyze social media images for climate discourse, even with moderate per-image accuracy.
• Magnifying What Matters: Attention-Guided Adaptive Rendering for Visual Text Comprehension — VLMs localize text evidence without full utilization; adaptive rendering improves comprehension.
• Understanding Pruning Regimes in Vision-Language Models Through Domain-Aware Layer Selection — Domain-aware pruning effectively reduces VLM depth by identifying and removing layers least critical to specific tasks.
• Fine-Tuning Qwen3.5 — Fine-tuning small vision-language models like Qwen3.5-0.8B on domain-specific datasets significantly improves specialized task performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.