MPDocBench-Parse: Benchmarking Practical Multi-page Document Parsing

2025-12-11 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

MPDocBench-Parse is a new benchmark designed to address the limitations of existing document parsing benchmarks, which are often inadequate for realistic multi-page scenarios. Developed by the University of Science and Technology of China and Tongyi Lab, Alibaba Group, it features 433 manually annotated documents across 3,246 pages, covering 15 diverse document types in both English and Chinese with varied layout styles. The benchmark supports document-level end-to-end evaluation and introduces a comprehensive protocol for assessing content fidelity and logical structure. This includes text, table, and formula recognition, truncated text and table merging, figure extraction, reading order, and heading hierarchy recovery. Experiments reveal that while current models excel at basic text extraction, they show clear limitations in semantic continuity integration, visual content parsing, and hierarchical structure recovery.

Key takeaway

For AI Engineers developing document parsing solutions for enterprise applications, recognize that current models, including general VLMs, still exhibit significant limitations in handling realistic multi-page documents. Your evaluation strategies should extend beyond basic text extraction to rigorously test semantic continuity, multimodal visual content extraction, and hierarchical structure recovery. Prioritize specialized models and research efforts that explicitly address these complex challenges to ensure robust real-world deployment.

Key insights

Practical multi-page document parsing requires benchmarks assessing semantic continuity, visual content, and hierarchical structure beyond basic text.

Principles

• Multi-page parsing needs end-to-end evaluation.
• Content fidelity includes semantic continuity and multimodal preservation.
• Logical structure requires reading order and hierarchy recovery.

Method

MPDocBench-Parse uses iterative annotation with model pre-annotation, human correction, and multi-dimensional verification for structure and content, including a Gemini-3-Pro refinement step for OCR.

In practice

• Evaluate models on truncated text/table merging.
• Assess figure extraction for visual content fidelity.
• Test heading hierarchy recovery for structural understanding.

Topics

• Document Parsing
• Multi-page Document Analysis
• Benchmark Datasets
• Vision-Language Models
• Information Extraction
• Semantic Structure Recovery

Code references

• Tongyi-Zhiwen/Qwen-Doc
• datalab-to/chandra

Best for: Research Scientist, NLP Engineer, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• RealDocBench: A Benchmark for Field-Level QA and Layout Understanding on Real-World Regulated Documents — RealDocBench offers a field-level, real-world benchmark exposing true document parsing system performance and operational trade-offs.
• Beyond extract_text: The Two Layers of a PDF That Drive RAG Quality — Effective RAG parsing requires understanding both PDF structural signals and page content, augmented by an LLM-generated semantic summary.
• Introducing ParseBench: The First Document Parsing Benchmark for AI Agents — ParsBench is a new benchmark for document parsing, emphasizing semantic correctness for AI agents.
• [R] IDP Leaderboard: Open benchmark for document AI across 16 VLMs, 9,000+ documents, 3 benchmark suites — An open benchmark evaluates 16 VLMs on document understanding, revealing tight performance margins among top models.
• ExStrucTiny: A Benchmark for Schema-Variable Structured Information Extraction from Document Images — ExStrucTiny is a new benchmark for structured information extraction from diverse, schema-variable document images.
• The PDF Paradox: Why Document Parsing Is Still Hard — And Why the Hybrid Stack Is Winning — PDF's fixed-layout design, while excellent for rendering, creates a significant parsing challenge for AI.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.