CLAY: Conditional Visual Similarity Modulation in Vision-Language Embedding Space

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, extended

Summary

CLAY is an adaptive similarity computation method designed to enhance image retrieval systems by modulating the embedding space of pretrained Vision-Language Models (VLMs) without additional training. This approach addresses the limitation of fixed similarity metrics by enabling text-conditional and multi-conditioned retrieval with fixed visual embeddings, ensuring high computational efficiency. The method constructs a manifold-aware textual subspace using singular value decomposition on text condition features, then projects visual features onto this space. Researchers also developed CLAY-EVAL, a synthetic dataset comprising 7,325 object images and 6,745 human images, for comprehensive evaluation under diverse conditional retrieval settings. Experiments demonstrate CLAY's superior retrieval accuracy and efficiency compared to prior methods.

Key takeaway

For Machine Learning Engineers building adaptive image retrieval systems, CLAY offers a training-free solution to incorporate diverse user conditions efficiently. You should consider integrating CLAY's similarity modulation to achieve high accuracy in multi-conditional scenarios without the computational overhead of recomputing database embeddings. This approach allows for more flexible and human-centric search experiences.

Key insights

CLAY adaptively modulates VLM embedding spaces for efficient, training-free, multi-conditional image retrieval.

Principles

• Decouple conditioning from visual feature extraction for efficiency.
• Model VLM embedding space as a hyperspherical manifold.
• Align visual and textual feature means for consistent tangent space mapping.

Method

CLAY constructs a manifold-aware textual subspace by applying SVD on logarithm-mapped text features, then projects pre-computed VLM visual features onto this subspace for conditional similarity calculation.

In practice

• Use CLAY for adaptive, multi-conditioned image search.
• Apply manifold-aware projection for VLM-based retrieval.

Topics

• Conditional Image Retrieval
• Vision-Language Models
• Embedding Space Modulation
• Multi-conditional Retrieval
• Synthetic Datasets
• Computational Efficiency

Best for: Research Scientist, AI Scientist, Computer Vision Engineer, Machine Learning Engineer

Related on AIssential

• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers — Finetuning multimodal embedding models on domain-specific data significantly boosts performance over general-purpose models.
• Exploring Topological Invariance in Semantic Embeddings — Preliminary research suggests topological properties of semantic embeddings may act as a cross-language and cross-model semantic invariant.
• Statistical Embeddings for Similarity, Retrieval, and Interpretable Alignment of Numeric Tabular Datasets — Statistical embeddings enable interpretable similarity and alignment of heterogeneous numeric tabular datasets for LLMs, preserving statistical context.
• Conceptualizing Embeddings: Sparse Disentanglement for Vision-Language Models — CEDAR disentangles vision-language model embeddings post-hoc, revealing compositional structure without increasing dimensionality, improving interpretability.
• SOTA Embedding Model for Agentic Workflows Now in Public Preview — Qwen3-Embedding-0.6B is a state-of-the-art 0.6B-class embedding model hosted on Databricks, outperforming larger models like OpenAI and Cohere's flagship offerings on MTEB leaderboards, providing top-…
• FlowCIR: Semantic Transport via Flow Matching for Zero-Shot Composed Image Retrieval — FlowCIR redefines ZS-CIR as conditional semantic transport, using flow matching for efficient, robust image retrieval.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.