KoViDoRe: A Benchmark for Korean Visual Document Retrieval

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Natural Language Processing · Depth: Expert, medium

Summary

KoViDoRe is a new benchmark for Korean visual document retrieval, introduced to address limitations in existing English-centric benchmarks and single-page Korean resources. Developed from publicly available Korean documents featuring diverse layouts like tables, figures, and multi-column structures, KoViDoRe aims to capture realistic multi-page retrieval scenarios. Its multi-stage data curation pipeline includes structured document parsing, synthetic query generation via summary-based and context-based strategies, and human-verified relevance mapping. Initial evaluations using KoViDoRe reveal that current multimodal retrieval models struggle with Korean visual documents, particularly when dealing with structured content and varied query types. To support model development, a large-scale training dataset, Ko-VDR Train Public, was also curated. Together, KoViDoRe and Ko-VDR Train Public offer a unified benchmark and training resource, presented at MAGMaR 2026, pages 54-80.

Key takeaway

For Machine Learning Engineers developing multimodal retrieval systems for non-English content, particularly Korean, you should recognize that existing models underperform on complex visual documents. Your focus should shift towards specialized benchmarks like KoViDoRe and training datasets such as Ko-VDR Train Public. Prioritize developing models capable of handling diverse layouts, structured content, and multi-page evidence aggregation to improve real-world performance in Korean visual document retrieval.

Key insights

Current multimodal retrieval models struggle with complex Korean visual documents, highlighting a critical need for specialized benchmarks and training data.

Principles

• Benchmarks need diverse, complex document structures.
• Multi-page evidence aggregation is crucial for realism.
• Language-specific datasets improve model performance.

Method

A multi-stage data curation pipeline involves structured document parsing, synthetic query generation (summary-based and context-based), and human-verified relevance mapping.

In practice

• Evaluate models on KoViDoRe for Korean VDR.
• Train models using Ko-VDR Train Public.
• Focus on structured content handling in VDR.

Topics

• Korean Visual Document Retrieval
• Multimodal Retrieval Benchmarks
• Document Layout Analysis
• Synthetic Query Generation
• KoViDoRe Dataset
• Ko-VDR Train Public

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• MKG-RAG-Bench: Benchmarking Retrieval in Multimodal Knowledge Graph-Augmented Generation — Retrieval quality is a critical bottleneck and strongly determines generation outcomes in multimodal knowledge graph RAG.
• VDE Bench: Evaluating The Capability of Image Editing Models to Modify Visual Documents — Image editing models require specialized benchmarks to accurately assess their performance on multilingual and densely textual visual documents.
• DocRetriever: A Plug-and-Play Framework for Multimodal Document Retrieval with Comprehensive Benchmark — DocRetriever improves multimodal document retrieval via layout-aware sparse embeddings and a generalizable, reasoning-augmented reranker.
• KVoiceBench, KOpenAudioBench, and KMMAU: Agent-Driven Korean Speech Benchmarks for Evaluating SpeechLMs — Multilingual SpeechLM evaluation requires language-specific benchmarks to reveal true performance gaps and task-specific weaknesses.
• Benchmarking Open-Source Layout Detection Models for Data Snapshot Extraction from Institutional Documents — Current open-source layout detection models fail to generalize for "data snapshot extraction" from complex institutional documents.
• SVFSearch: A Multimodal Knowledge-Intensive Benchmark for Short-Video Frame Search in the Gaming Vertical Domain — Specialized benchmarks are crucial for evaluating MLLMs in knowledge-intensive, visually ambiguous short-video domains.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.