Show Me Examples: Inferring Visual Concepts from Image Sets

2026-07-17 · Source: Apple Machine Learning Research · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, quick

Summary

The July 2026 paper "Show Me Examples: Inferring Visual Concepts from Image Sets" addresses a critical limitation in current Vision-language models (VLMs): their inability to infer shared concepts from sets of example images and apply them to new inputs. These models often ignore visual context or produce biased generations. Researchers introduce Visual Concept Inference from Sets (VICIS), a new task where a model must generate images that preserve a concept defined by a small context set while remaining consistent with a query image. Existing VLMs perform poorly on VICIS. To overcome this, the paper proposes a novel training framework and architecture that learns to infer visual concepts from image sets and extract concept-specific embeddings. Experiments on synthetic and large-scale ImageNet/WordNet data demonstrate that their model generates more accurate and diverse outputs, generalizing effectively to unseen concepts and modalities like sketches.

Key takeaway

For Machine Learning Engineers developing advanced VLM applications, this research highlights a significant gap in current VLM capabilities regarding visual concept inference from example sets. Existing models often fail to leverage visual context effectively. You should consider integrating concept-specific embedding extraction and set-based learning into your model architectures. This approach can improve output accuracy and diversity in image generation tasks, particularly when generalizing to novel visual concepts or modalities like sketches, moving beyond biased or context-ignoring generations.

Key insights

VLMs struggle with visual concept inference from image sets; a new framework improves this capability.

Principles

• VLMs often ignore visual context in set-based inference.
• Concept-specific embeddings enhance visual concept learning.
• Generalization to unseen concepts and modalities is achievable.

Method

A training framework and architecture learn to infer visual concepts from image sets and extract concept-specific embeddings from query images to generate new outputs.

In practice

• Generate images preserving a concept from examples.
• Apply inferred concepts to new query images.
• Extend concept inference to modalities like sketches.

Topics

• Computer Vision
• Vision-Language Models
• Visual Concept Inference
• Image Generation
• Machine Learning Architectures
• Concept Embeddings

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Show Me Examples: Inferring Visual Concepts from Image Sets — VLMs struggle with visual concept inference from image sets; a new framework addresses this gap.
• Show Me Examples: Inferring Visual Concepts from Image Sets — Vision-language models require explicit training to infer and apply visual concepts from image sets for robust visual in-context learning.
• VLCE: A Knowledge-Enhanced Framework for Image Description in Disaster Assessment — Integrating external knowledge graphs into VLMs significantly enhances domain-specific image descriptions and reduces factual errors.
• Video Scene Graph Generation Using VLMs — VLMs enable open-vocabulary video scene graph generation, overcoming fixed ontology limitations for richer video understanding.
• Segmentation before Answering: Pixel Grounding for MLLM Visual Reasoning — Pixel-level segmentation significantly enhances MLLM visual reasoning by precisely isolating regions of interest, outperforming bounding box methods.
• Beyond Scalar Distances: Semantic Attribute Gradients from Frozen MLLMs for Visual Embeddings — SAGA uses frozen MLLMs to provide attribute-resolved supervision for vision encoders, moving beyond scalar distances.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Apple Machine Learning Research.