Show Me Examples: Inferring Visual Concepts from Image Sets

2026-07-23 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, extended

Summary

Visual Concept Inference from Sets (VICIS) is a new task introduced to evaluate vision-language models' ability to reason from purely visual context. Leading vision-language models, including BAGEL and ILLUME+, perform poorly on VICIS, often ignoring visual context or defaulting to biased generations. The task requires models to infer a shared concept from a small set of example images and apply it to a query image, generating new images that preserve the context-defined concept while remaining consistent with the query. A novel training framework and architecture are proposed to address this gap, comprising a Set Learner, Instantiation Stage, and Diffusion Model. This model, trained on synthetic and large-scale ImageNet/WordNet data using 16 H100 GPUs for 24 hours (765M parameters), generates more accurate (46.38% per concept) and diverse (0.81 diversity score) outputs, generalizing to unseen concepts and modalities like sketches.

Key takeaway

For AI Scientists and Machine Learning Engineers developing generative vision models, this research highlights that leading vision-language models struggle with inferring visual concepts from example image sets. You should consider specialized architectures and training frameworks, such as the VICIS approach, to enable robust visual in-context learning. This is critical for applications requiring nonverbal concept specification or generalization to modalities like sketches, moving beyond reliance on text-only instructions.

Key insights

Vision-language models require explicit training to infer and apply visual concepts from image sets for robust visual in-context learning.

Principles

• Visual in-context learning demands concept inference from image sets.
• Auxiliary groupings like WordNet hierarchies enable scalable training data construction.
• Concept-specific embeddings facilitate structured reasoning and cross-modal generalization.

Method

The proposed method uses a Set Learner (ViT) to identify concept directions from image set embeddings, an Instantiation Module to project query embeddings onto these directions, and a Diffusion Model to generate images conditioned on the projected query, trained end-to-end with a rectified flow objective.

In practice

• Construct training data using ImageNet/WordNet hierarchies for concept inference.
• Employ a Set Learner to derive concept-specific embedding directions.
• Condition generative models on concept-specific query embeddings.

Topics

• Visual Concept Inference
• Vision-Language Models
• In-Context Learning
• Image Generation
• Diffusion Models
• ImageNet WordNet Hierarchy
• Visual Reasoning

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Show Me Examples: Inferring Visual Concepts from Image Sets — VLMs struggle with visual concept inference from image sets; a new framework improves this capability.
• Show Me Examples: Inferring Visual Concepts from Image Sets — VLMs struggle with visual concept inference from image sets; a new framework addresses this gap.
• Vision Models in AI: How Machines Learn to See the World — Vision models have evolved from CNNs to multimodal VLMs, enabling machines to interpret and generate visual data with semantic understanding.
• Scalable Visual Pretraining for Language Intelligence — Visual pretraining on documents directly utilizing visual cues consistently outperforms text-only methods for language intelligence.
• Harnessing Generative Image Models for Training-Free Primitive Shape Abstraction — Training-free 3D shape abstraction leverages generative image models for semantic part segmentation and classical optimization for primitive fitting.
• BabyVision: Visual Reasoning Beyond Language — MLLMs lack fundamental visual primitives, struggling with basic tasks solvable by young children due to a "verbalization bottleneck."

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.