CMDR: Contextual Multimodal Document Retrieval

2026-07-08 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, long

Summary

CMDR and CMDR-Bench introduce a new task and benchmark for Contextual Multimodal Document Retrieval, addressing the limitation of existing methods that overlook document context. CMDR-Bench features 800 human-annotated queries across four categories (Text Completion, Coreference Resolution, Structured Understanding, Multi-hop Reasoning) and 255 documents averaging 183.5 pages from six diverse domains. To tackle this, CMDR-Embed is proposed, a contextual multimodal embedding framework that jointly encodes multiple pages using a chunk-then-split strategy. It is trained with CMCL, a novel contrastive learning objective balancing contextual modeling and page-level discriminability. Experimental results demonstrate CMDR-Embed significantly outperforms non-contextual embeddings, including those augmented with rerankers like Qwen3-VL Reranker, with minimal computational overhead. OCR quality showed limited impact on text-based retrievers, underscoring the importance of context.

Key takeaway

For Machine Learning Engineers developing multimodal retrieval systems, you should prioritize context-aware embedding models. Your current independent page encoding methods likely miss crucial cross-page dependencies, leading to suboptimal retrieval. Consider adopting joint encoding strategies like CMDR-Embed and training with contrastive learning objectives that balance context and page discriminability. This approach improves accuracy for complex queries without significant computational overhead, enhancing your RAG applications.

Key insights

Contextual multimodal document retrieval requires modeling cross-page dependencies for accurate information retrieval.

Principles

• Document context is crucial for resolving ambiguities in multimodal retrieval.
• Jointly encoding multiple pages captures cross-page contextual relationships.
• Balancing context modeling with page-level discriminability is key for embeddings.

Method

CMDR-Embed uses a chunk-then-split strategy: consecutive pages are jointly encoded by an LVLM, then representations are split into page-level embeddings. CMCL trains this by incorporating context-aware hard negatives.

In practice

• Use CMDR-Bench for evaluating context-aware multimodal retrieval.
• Implement joint encoding for multi-page document embeddings.
• Employ contrastive learning with context-aware hard negatives.

Topics

• Multimodal Document Retrieval
• Contextual Embeddings
• Contrastive Learning
• Document RAG
• CMDR-Bench
• Large Vision-Language Models

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• What is Multimodal RAG? Unlocking LLMs with Vector Databases — Multimodal RAG enables LLMs to reason over diverse data types by integrating non-textual information into retrieval and generation.
• DocRetriever: A Plug-and-Play Framework for Multimodal Document Retrieval with Comprehensive Benchmark — DocRetriever improves multimodal document retrieval via layout-aware sparse embeddings and a generalizable, reasoning-augmented reranker.
• ELVA: Exploring Ranking-Driven Universal Multimodal Retrieval — ELVA mitigates "grain blindness" in multimodal retrieval by ranking negative samples based on similarity, not just binary classification.
• Embed the world: Multimodal AI for searchable aerial imagery at scale — Multimodal AI, combining image embeddings and LLM captions, enables efficient natural-language search over complex aerial imagery.
• From Scenes to Elements: Multi-Granularity Evidence Retrieval for Verifiable Multimodal RAG — Element-level visual evidence retrieval significantly enhances multimodal RAG verifiability and performance.
• Manufacturing intelligence with Amazon Nova Multimodal Embeddings — Multimodal embeddings significantly enhance information retrieval from visually rich technical documents compared to text-only methods.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.