Explaining is Harder Than Predicting Alone: Evaluating Concept-based Explanations of MLLMs as ICL Visual Classifiers

2026-04-30 · Source: cs.MA updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Explainable AI · Depth: Expert, extended

Summary

A study systematically evaluated the concept-based explainability of four state-of-the-art Multimodal Large Language Models (MLLMs) under few-shot in-context learning (ICL) for image classification. Using five conditions of increasing formal rigor, from baseline classification to Description Logics (DL) axiom generation, and an independent LLM-as-a-judge pipeline, researchers found that demanding explanations degrades predictive accuracy. Accuracy dropped monotonically from 93.8% (no explanation) to 90.1% (DL Axioms). While MLLMs excel at visual classification, they lack the specific instruction-tuning required for formal, machine-verifiable explainability. However, explanation quality, particularly "Local Discriminativeness," strongly correlated with correct predictions when models articulated class-discriminative visual features.

Key takeaway

For AI Scientists developing or deploying MLLMs, recognize that current models are not inherently optimized for formal, machine-verifiable explanations. Forcing complex explanation generation, such as Description Logics axioms, can reduce classification accuracy. Instead, focus instruction-tuning efforts on improving "Local Discriminativeness" to enhance both explainability and predictive reliability, and consider simpler explanation formats for practical applications.

Key insights

MLLMs struggle with formal concept-based explanations, degrading prediction accuracy, despite strong visual classification.

Principles

• Demanding formal explanations from MLLMs reduces predictive accuracy.
• Classification accuracy does not guarantee explanation quality.
• Local discriminativeness strongly predicts correct classifications.

Method

A systematic evaluation uses five explanation conditions, from natural language to Description Logics axioms, assessed by an LLM-as-a-judge pipeline with nine XAI metrics.

In practice

• Use LLM-as-a-judge for scalable XAI evaluation.
• Prioritize Local Discriminativeness in XAI metrics.
• Consider simpler explanation formats (E3, E4).

Topics

• In-Context Learning
• Multimodal LLMs
• Explainable AI
• Concept-based Explanations
• Description Logics
• Visual Classification
• LLM-as-a-Judge

Code references

• meta-llama/llama-models

Best for: Research Scientist, Computer Vision Engineer, AI Scientist

Related on AIssential

• Decoding the Multimodal Maze: A Systematic Review on the Adoption of Explainability in Multimodal Attention-based Models — Multimodal XAI evaluation lacks standardization, hindering interpretable and responsible AI development.
• Symbolic Grounding Reveals Representational Bottlenecks in Abstract Visual Reasoning — Symbolic input dramatically improves abstract visual reasoning in LLMs, indicating representation, not reasoning, is the VLM bottleneck.
• Why Multimodal LLMs Fail at Retrieval, Reasoning-Enhanced Sequential Modeling for Industrial Recommendation, and More! — MLLMs struggle with multimodal retrieval due to modality imbalance and homogenized embeddings, while LLM reasoning enhances recommender systems.
• Using Large Language Models and Knowledge Graphs to Improve the Interpretability of Machine Learning Models in Manufacturing — Integrating KGs with LLMs can significantly improve ML model interpretability by providing structured, context-rich explanations.
• Improving AI models’ ability to explain their predictions — Extracting inherent model concepts yields more accurate and interpretable AI explanations than predefined human concepts.
• A Definition of Good Explanations and the Challenges Explaining LLM Outputs — Good AI explanations combine counterfactuals with the interlocutor's prior beliefs to address LLM explainability challenges.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.MA updates on arXiv.org.