See & Sniff: Learning Visuo-Olfactory Representations

2026-06-25 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, quick

Summary

A new research initiative introduces SmellNet-V, a scalable visuo-olfactory dataset, and See & Sniff, a self-supervised framework for learning joint visuo-olfactory representations. SmellNet-V addresses the lack of paired visuo-olfactory data by leveraging the insight that odor identity remains largely invariant to visual transformations within a semantic category. This enables the synthetic pairing of smell-only samples with semantically aligned web images, creating a cross-modal benchmark without costly co-collection. The See & Sniff framework utilizes dense local alignment to learn these representations, naturally generating smell saliency maps for spatial grounding of odor sources. It also establishes a pixel-level smell localization task and benchmark. This approach improves smell classification from smell alone by 7% over smell-only baselines and extends to cross-modal retrieval and smell localization, pioneering visuo-olfactory learning in multimodal perception.

Key takeaway

For Machine Learning Engineers developing multimodal perception systems, this work introduces a viable path for integrating olfaction. You should consider utilizing the principle of semantic invariance to synthetically expand scarce cross-modal datasets, as demonstrated by SmellNet-V. This approach allows you to build robust visuo-olfactory models, improving classification and enabling spatial smell localization, even without expensive co-collected data.

Key insights

Odor identity's invariance to visual transformations enables synthetic visuo-olfactory data generation and joint representation learning.

Principles

• Odor identity is invariant to visual transformations within a semantic category.
• Self-supervised learning can align distinct modalities without direct co-collection.
• Dense local alignment facilitates spatial grounding of sensory inputs.

Method

The See & Sniff framework learns joint visuo-olfactory representations using dense local alignment, producing smell saliency maps. It utilizes synthetically paired smell-only samples with web images from the SmellNet-V dataset.

In practice

• Synthetically generate cross-modal datasets from unimodal sources.
• Develop models for smell classification and localization.
• Ground odor sources spatially using smell saliency maps.

Topics

• Visuo-Olfactory Learning
• Multimodal Perception
• Self-Supervised Learning
• SmellNet-V Dataset
• Smell Localization
• Cross-Modal Retrieval

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models — Self-evolving LMMs often under-condition visually; VISE uses invariance rewards to enforce spatial and semantic consistency.
• FoodSense: A Multisensory Food Dataset and Benchmark for Predicting Taste, Smell, Texture, and Sound from Images — FoodSense dataset enables AI models to infer multisensory food experiences from images, bridging cognitive science and multimodal AI.
• From Visual to Multimodal: Systematic Ablation of Encoders and Fusion Strategies in Animal Identification — Multimodal fusion of visual and synthetic text features significantly enhances animal re-identification accuracy and reliability.
• Visual-OPSD: Cross-Modal On-Policy Self-Distillation for Efficient Unified Multimodal Reasoning — The generation pathway of visual thoughts encodes reasoning that can be distilled for efficient, text-only multimodal models.
• NeuralMUSIC: A Hybrid Neural-Subspace Framework for Robot Sound Source Localization — A hybrid neural-subspace framework enhances robot sound source localization by combining deep learning with classical signal processing.
• Sensor-Conditioned Representation Learning via Scene-Relevant Observation Quotients — Sensor-conditioned representations require evaluation beyond predictive utility, focusing on preserving sensing-justified scene distinctions in their latent geometry.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.