EduArt: An educational-level benchmark for evaluating art history knowledge in large language models

2026-07-02 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Computer Vision & Pattern Recognition · Depth: Expert, quick

Summary

EduArt is a new educational-level benchmark designed to evaluate art-historical knowledge and visual reasoning in multimodal large language models. It comprises 871 human-authored questions sourced from Italian secondary-school exercises and US Advanced Placement Art History exams, covering two languages and seven formats, including multiple choice, in-text word placement, and error identification. Twelve models from six provider families were assessed under both default answer-only and motivation conditions. The benchmark demonstrated strong psychometric properties, with a mean discrimination of 0.514 and 82.3 percent good discriminators. While six models achieved near-ceiling accuracy on multiple-choice questions, performance significantly dropped on other formats; for instance, models exceeding 94 percent on multiple choice fell to 23.9 percent on open completion (Claude Opus 4.6) and 6.2 percent on error identification (Claude Sonnet 4.6). The motivation condition generally decreased accuracy. These results indicate a distinction between possessing art-historical knowledge and the ability to deploy it, suggesting single-format benchmarks overstate model capabilities.

Key takeaway

For AI Scientists and NLP Engineers developing or deploying multimodal LLMs for art-historical scholarship, you must move beyond simple multiple-choice evaluations. Your models' high scores on recognition tasks do not reflect their ability to produce or manipulate content, as evidenced by significant performance drops on open completion and error identification. To ensure responsible use, rigorously test your models across diverse question formats and conditions, including those requiring justification, to accurately map their capability profiles.

Key insights

Art-historical knowledge and its deployment are distinct LLM capabilities, often overestimated by single-format benchmarks.

Principles

• General benchmarks obscure discipline-specific LLM behavior.
• Single-format evaluations overstate LLM capabilities.
• Psychometrically robust benchmarks are crucial for LLM assessment.

Method

EduArt evaluates multimodal LLMs using 871 human-authored questions from secondary-school and AP exams, spanning two languages and seven formats, with answer-only and motivation conditions.

In practice

• Use diverse question formats for comprehensive LLM evaluation.
• Assess LLMs on content production, not just selection.
• Consider psychometric properties for benchmark design.

Topics

• Large Language Models
• Multimodal LLMs
• Art History
• Benchmark Evaluation
• Psychometrics
• Question Formats

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• CARTE: A Benchmark for Mapping Language Model Knowledge Across France — LLMs exhibit significant knowledge gaps in fine-grained, geographically specific regional understanding, particularly within countries.
• AttuneBench: A Conversation-Based Benchmark for LLM Emotional Intelligence — LLM emotional intelligence is a multi-faceted capability, requiring distinct evaluation metrics beyond simple emotion recognition.
• AGORA: An Archive-Grounded Benchmark for Agentic Workplace Document Reasoning — Archive-grounded reasoning in large, messy document collections remains a significant challenge for LLM agents, even with advanced benchmarks.
• Faithful, Enriched, and Precise: Benchmarking Natural-Science Illustration Generation by T2I models — FEPBench offers a fine-grained, atom-set-based benchmark for T2I models generating scientific illustrations, revealing current limitations in text, reasoning, and precision.
• The Anatomy of an LLM Benchmark — Effective LLM benchmarking requires continuous evolution, rigorous data quality, and dynamic evaluation methods to counter rapid model saturation.
• Most Influential ArXiv (Computation and Language) Papers (2026-04 Version) — Citation-based ranking reveals key trends in NLP research, emphasizing LLM reasoning, efficiency, and multimodal capabilities.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.