Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations

2026-05-19 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Explainable AI · Depth: Expert, extended

Summary

Vision-plus-Human-guided Concept Bottleneck Models (VH-CBMs) are a hybrid approach designed to improve the interpretability and applicability of neural classifiers by combining Vision-Language Models (VLMs) with minimal expert annotations. Traditional Concept Bottleneck Models (CBMs) require extensive, high-quality concept annotations, which are often unavailable. While VLM-guided CBMs (VLM-CBMs) address this by using weak supervision from VLMs, this can lead to less accurate and less interpretable concepts. VH-CBMs introduce Gaussian Processes (GPs) in the VLM's embedding space to propagate expert supervision from as little as 1% of annotated data, enhancing concept accuracy, calibration, and disentanglement. Empirical evaluations on datasets like Shapes3d, CelebA, CUB, and Derma demonstrate that VH-CBMs significantly outperform VLM-CBMs in concept accuracy and calibration, while maintaining competitive task performance, even surpassing fully supervised CBMs in some cases.

Key takeaway

For Research Scientists developing interpretable AI models, VH-CBMs offer a compelling solution to the interpretability-applicability trade-off. Your teams can achieve substantially more accurate and calibrated concepts with minimal expert annotation (e.g., 1% of data), which is critical for high-stakes applications. Consider integrating Gaussian Processes into your VLM-CBM pipelines to leverage both the broad applicability of VLMs and the precision of human supervision, potentially reducing annotation costs through active learning strategies.

Key insights

VH-CBMs enhance concept accuracy and interpretability in CBMs by integrating VLM embeddings with minimal expert annotations via Gaussian Processes.

Principles

• Interpretability and applicability often present a trade-off in CBM architectures.
• VLM embedding spaces encode useful global information despite local annotation inaccuracies.
• Bayesian models like GPs improve concept calibration and enable active learning.

Method

VH-CBMs use a VLM for embeddings, then train per-concept Gaussian Processes on a small, expert-annotated subset. These GPs propagate supervision and estimate concept activations, which feed into a linear inference layer for task prediction.

In practice

• Annotate as little as 1% of data to significantly boost concept accuracy.
• Employ GP uncertainty estimates for efficient active learning of concepts.
• Utilize CLIP or DINO backbones for VLM embeddings.

Topics

• VH-CBM
• Concept Bottleneck Models
• Vision-Language Models
• Gaussian Processes
• Concept Accuracy

Code references

• osmr/imgclsmob

Best for: Research Scientist, Computer Vision Engineer, AI Scientist, Machine Learning Engineer

Related on AIssential

• Assessing Reliability of Symbol Detection in Concept Bottleneck Models — CBMs' high accuracy doesn't guarantee reliable symbol detection; independent component swapping reveals shortcut reliance.
• Prototype-Grounded Concept Models for Verifiable Concept Alignment — PGCMs enhance CBM interpretability by grounding abstract concepts in verifiable visual prototypes, enabling direct inspection and intervention.
• Bridging Vision and Language Concepts through Optimal Transport Semantic Flow — OTF-CBM uses optimal transport flow to dynamically align visual and textual concepts, improving interpretability and accuracy in CBMs.
• Improving AI models’ ability to explain their predictions — Extracting inherent model concepts yields more accurate and interpretable AI explanations than predefined human concepts.
• LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs — LatentLens reveals visual tokens are highly interpretable by comparing them to contextualized text embeddings, not just static vocabulary.
• Concept Flow Models: Anchoring Concept-Based Reasoning with Hierarchical Bottlenecks — Concept Flow Models (CFMs) employ hierarchical concept-driven decision trees to mitigate information leakage and enhance interpretability in concept-based reasoning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.