DrawingVQA: A Real-World Benchmark for Multi-Depth Visual-Textual Reasoning on Construction Drawings

2026-07-16 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Advanced, quick

Summary

DrawingVQA is introduced as the first benchmark specifically designed to evaluate multimodal large language models (MLLMs) on real-world construction drawings, a critical media in architecture and civil engineering. This benchmark addresses the unique complexity of construction drawings, which fuse abstract geometry, symbolic notation, tabular data, annotations, and domain-specific text. DrawingVQA comprises 33 "Issued for Construction" drawings and 92 expertly curated question-answer pairs, categorized into three reasoning depths: perceptual understanding, contextual interpretation, and domain-expert reasoning. A dual categorization framework is presented to analyze MLLM performance across seven construction-engineering and four MLLM capability dimensions. Initial evaluations of leading MLLMs reveal a substantial performance gap compared to human experts, particularly at higher reasoning depths, highlighting the need for advancements in domain-specialized multimodal reasoning for engineering workflows.

Key takeaway

For AI Scientists and Machine Learning Engineers developing MLLMs for architecture, engineering, and construction, you should recognize that current models exhibit a substantial performance gap on real-world construction drawings. Your development efforts must prioritize domain-specialized multimodal reasoning, particularly focusing on improving contextual interpretation and domain-expert reasoning capabilities. This benchmark indicates a critical need to advance MLLM understanding beyond perceptual tasks to integrate AI effectively into complex engineering workflows.

Key insights

MLLMs significantly underperform experts on multi-depth reasoning for complex construction drawings.

Principles

• Construction drawings demand multi-modal, multi-depth reasoning.
• Current MLLMs struggle with domain-expert interpretation.
• Benchmarks must map engineering workflows to AI competencies.

Method

DrawingVQA evaluates MLLMs using 33 "Issued for Construction" drawings and 92 Q&A pairs, categorized by three reasoning depths and a dual framework.

In practice

• Benchmark MLLMs using DrawingVQA for engineering tasks.
• Prioritize MLLM development for domain-expert reasoning.

Topics

• DrawingVQA
• Multimodal LLMs
• Construction Drawings
• Visual Question Answering
• Engineering AI
• Domain-Specific Reasoning

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• MM-CondChain: A Programmatically Verified Benchmark for Visually Grounded Deep Compositional Reasoning — Deep compositional reasoning in MLLMs remains a significant challenge, especially with increasing complexity.
• How Multimodal Large Language Models See, Think, and Reason — MLLMs integrate visual perception into transformers by converting image patches into tokens for sequential processing and reasoning.
• ExpertVerse: A General-Purpose Benchmark for Expert-Level Reasoning in Knowledge-Intensive Visual Synthesis — ExpertVerse benchmarks knowledge-intensive visual reasoning, revealing current multimodal generative models' critical deficits.
• ISO-Bench: Benchmarking Multimodal Causal Reasoning in Visual–Language Models through Procedural Plans — Multimodal models struggle with causal reasoning, achieving only 0.57 F1 on ISO-Bench, far below human performance.
• MechVQA: Benchmarking and Enhancing Multimodal LLMs on Comprehensive Mechanical Drawing Understanding — MechVQA and MechVL enhance MLLM understanding of complex mechanical drawings through specialized data and multi-stage training.
• Segmentation before Answering: Pixel Grounding for MLLM Visual Reasoning — Pixel-level segmentation significantly enhances MLLM visual reasoning by precisely isolating regions of interest, outperforming bounding box methods.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.