See & Sniff: Learning Visuo-Olfactory Representations

Source: arXiv:2606.27307 · Published 2026-06-25 · By Seongyu Kim, Seungwoo Lee, Hyeonggon Ryu, Joon Son Chung, Arda Senocak

TL;DR

This work addresses the longstanding challenge of integrating olfaction into multimodal AI systems, which has been limited by the scarcity of paired visuo-olfactory data. The authors introduce SmellNet-V, a scalable visuo-olfactory dataset created by synthetically pairing smell-only samples from the existing SmellNet chemical sensor dataset with semantically aligned, in-the-wild web images. This leverages the insight that odor identity is largely invariant within a semantic category despite visual variability, thus enabling scalable cross-modal learning without costly co-collection.

Building on SmellNet-V, the authors propose See & Sniff, a self-supervised framework that learns joint visuo-olfactory representations via a dense local alignment contrastive objective. This approach also naturally produces smell saliency maps localizing odor sources in images. The framework achieves a 7% improvement over smell-only baselines for odor classification from smell signals alone, generalizes to effective cross-modal retrieval, and enables the novel task of pixel-level smell source localization. Together, these contributions pioneer visuo-olfactory multimodal learning for machine perception.

Key findings

• See & Sniff improves smell classification accuracy by 7% over the smell-only ScentFormer baseline (from ~50.6% to 57.7% accuracy) on the SmellNet-Test dataset (Table 1).
• Dense local alignment with modality-specific aligners outperforms both global alignment and models without aligners for smell classification, cross-modal retrieval, and localization.
• See & Sniff achieves the highest Recall@1 for cross-modal retrieval in both smell->vision (56.14%) and vision->smell (63.2%) directions on SmellNet-V-Test, outperforming global alignment and a Global-CLIP baseline (Table 2).
• For smell localization, See & Sniff achieves mAP = 0.8362 and mIoU = 0.6456, significantly outperforming vision-only baselines (DINOv3 Att. Map mAP=0.6596, mIoU=0.5157) and cascaded approaches (SmellNet + SAM3 mIoU=0.3214) (Table 4).
• Joint end-to-end visuo-olfactory learning surpasses cascaded pipelines that separately classify odors then segment images, highlighting the value of dense cross-modal alignment.
• Smell classification improvements are consistent across all major ingredient families (fruits, herbs, nuts, spices, vegetables), with strongest gains in fruits, nuts, and vegetables (Fig 3).
• Interactive smell localization experiments confirm that the framework can dynamically ground different odors within the same scene, achieving an IIoU of 0.35 vs 0.06 for cascaded baseline (Table 5).
• Visual supervision during training improves unimodal olfactory embeddings, as evidenced by superior Smell->Smell retrieval performance relative to smell-only baselines (Table 3).

Threat model

n/a - This paper focuses on visuo-olfactory multimodal representation learning rather than adversarial threat modeling or security considerations. The assumed setting is benign training with synthetically paired data and no malicious actors.

Methodology — deep read

1. Threat Model & Assumptions: The adversary model is not explicitly defined as the paper focuses on multimodal representation learning rather than security. The framework assumes paired visuo-olfactory data for training, which is synthetically constructed under the assumption that odor identity is invariant to visual transformations within ingredient categories. Visual semantic alignment is used as a proxy for natural co-occurrence.

2. Data: The core dataset is SmellNet [12], a large-scale real-world olfactory dataset comprising roughly 180,000 time-series sensor samples collected at 1 Hz over 6 sessions from 50 food and natural ingredients. Each ingredient has diverse smell recordings using 6 sensor channels capturing chemical compounds relevant to odors. The authors extend SmellNet by collecting thousands of web images via a large language model (LLM)-generated descriptive querying process, retrieving context-rich, in-the-wild photos per ingredient. Images undergo multi-stage filtering including CLIP prompt-based verification, watermark detection, and human screening to ensure semantic and visual quality. Synthetic pairing is then done at the ingredient category level by randomly matching smell snippets ('sniff units') with semantically corresponding images, resulting in the SmellNet-V dataset with 5,411 sniff-image pairs for training and aligned test splits.

Sniff units are formed by segmenting continuous odor time series into windows (typically 50 timesteps with 25 stride) to emulate discrete olfactory sampling episodes.

3. Architecture & Algorithm: See & Sniff uses a two-stream model with separate visual and olfactory encoders. Visual backbone is DINOv3-Small pretrained and frozen. Olfactory branch uses a 4-layer transformer similar to ScentFormer, trained from scratch. Each encoder is followed by a lightweight modality-specific aligner -- a channel-wise layer norm plus 1x1 convolution for vision, and a 2-layer MLP with residual connections for smell -- projecting into a shared embedding space.

The core novelty is a dense local alignment contrastive objective. For a sniff-image pair, the smell features are temporally average pooled to vector ¯fo, and spatial visual feature maps f_v are extracted. A similarity map M is computed by inner products of ¯fo and each spatial visual feature vector, with the final similarity score obtained by max pooling over spatial locations. Contrastive loss then attracts positive pairs and repels negatives symmetrically between modalities.

4. Training Details: Training occurs on 224x224 images and fixed-length sniff units (50 timesteps at 1Hz), using a batch size of 64 on an NVIDIA RTX A5000 GPU. A first-order temporal difference with lag p=25 is applied to odor signals to emphasize changes. Data pairs are dynamically re-sampled each epoch by cycling images within ingredient categories to promote robustness. The vision encoder backbone is frozen to preserve pretrained representations, with only aligners and smell transformer trained.

5. Evaluation Protocol: Evaluation tasks include smell classification (linear probe trained on frozen smell encoder, tested on SmellNet smell-only splits), cross-modal retrieval (Recall@K on SmellNet-V test), and smell localization (mAP and mIoU on a newly annotated pixel-level localization benchmark, SmellNet-V-Source). Baselines include smell-only ScentFormer, global alignment variants without local alignment, a Global-CLIP baseline combining frozen CLIP vision features with smell encoder, as well as vision-only and cascaded pipeline methods. Multi-family breakdowns and ablation comparisons isolate effects of local alignment and aligner modules. An interactive localization benchmark tests grounding of multiple smells in single images.

6. Reproducibility: The SmellNet dataset [12] is publicly available; SmellNet-V is synthetically constructed using that data and publicly scraped images, but image datasets and prompts are not fully disclosed here. Source code or trained weights are not explicitly mentioned in the truncated text, so open reproducibility status is unclear. Details on prompt formulations and filtering are provided in supplemental materials.

Example end-to-end: For training, an odor time series from the SmellNet dataset is segmented into a 50-timestep sniff unit; a diverse web image semantically matching that ingredient category is paired. The smell transformer and frozen vision backbone extract modality features; aligners project these to a shared latent space. Dense similarity maps measure local feature correspondence. Cross-modal contrastive loss is computed against other negative pairs in the batch. Over epochs, the model learns joint embeddings enabling downstream smell classification, retrieval, and localization.

Technical innovations

• SmellNet-V: A synthetic visuo-olfactory dataset constructed by pairing odor snippets with semantically matched in-the-wild images leveraging odor invariance within categories.
• See & Sniff: A self-supervised framework for joint visuo-olfactory representation learning using dense local alignment rather than global pooling, enabling spatial grounding of smells.
• A dense similarity map-based cross-modal contrastive loss that aligns temporally pooled olfactory features to spatial visual feature maps via max pooling.
• Introduction of the pixel-level smell localization task and benchmark, demonstrating the ability to spatially ground odor sources in images.

Datasets

• SmellNet — ~180,000 olfactory time-series samples — real-world chemical sensor data from 50 ingredients
• SmellNet-V — 5,411 paired sniff-image samples — synthetic visuo-olfactory dataset constructed from SmellNet and in-the-wild web images
• SmellNet-Test — 1,083 sniff-only samples — official smell test split
• SmellNet-V-Test — same number of sniff samples as SmellNet-Test with paired images — for cross-modal retrieval
• SmellNet-V-Source — subset of SmellNet-V-Test with manually annotated pixel-level ingredient masks — for smell localization
• SmellNet-V-InteractiveSource — 100 multi-ingredient images with segmentation masks — for interactive smell localization

Baselines vs proposed

• ScentFormer (smell-only baseline): smell classification accuracy = 50.6% vs See & Sniff: 57.7% accuracy (7.1% absolute improvement) (Table 1)
• Global-CLIP (frozen CLIP vision + olfactory encoder with global contrastive loss): smell classification accuracy = 53.19% vs See & Sniff: 57.71% (Table 1)
• Ours-Global (global alignment variant): smell classification accuracy = 53.74% vs See & Sniff (local alignment + aligner) = 57.71% (Table 1)
• Cross-modal Retrieval R@1, Smell->Vision: Global-CLIP = 53.28%, See & Sniff = 56.14% (Table 2)
• Cross-modal Retrieval R@1, Vision->Smell: Global-CLIP = 56.90%, See & Sniff = 63.2% (Table 2)
• Smell Localization mIoU: Vision-only DINOv3 Att. Map = 0.5157; cascaded SmellNet + SAM3 = 0.3214; See & Sniff (local alignment) = 0.6456 (Table 4)
• Smell Localization mAP: Vision-only DINOv3 Att. Map = 0.6596; See & Sniff = 0.8362 (Table 4)
• Smell->Smell Retrieval R@1: SmellNet baseline = 52.54%, See & Sniff = 69.44% (Table 3)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.27307.

Fig 1: What can See & Sniff do? We show that See & Sniff, the framework that

Fig 2 (page 1).

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• Synthetic pairing assumes odor identity invariance within semantic categories, which may not hold for subtle visual variants affecting smell.
• No evaluation under adversarial conditions or with real co-collected paired visuo-olfactory data to validate alignment in the wild.
• Smell localization depends on segmentation masks generated by interactive SAM annotation, which may have annotation noise or domain gaps.
• Limited size of SmellNet-V (5,411 pairs) restricts diversity compared to truly co-collected real paired datasets.
• Vision encoder is frozen from a vision-only pretrained model, potentially limiting full joint multimodal adaptation.
• No testing on non-food or more chemically diverse odor categories beyond the 50 food/natural ingredients.
• Lack of open-source code or full details about image scraping and filtering reduces reproducibility.

Open questions / follow-ons

• Can real paired visuo-olfactory data collection validate and improve performance beyond synthetic pairing assumptions?
• How well do these methods generalize to non-food or complex environmental odors where visual-olfactory correlations are weaker or less consistent?
• What improvements or new tasks become possible with end-to-end jointly trained vision and olfactory backbones rather than frozen vision models?
• Can dense local alignment be extended to temporal localization of odors in video streams or multimodal sensor fusion with other sensory modalities?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this work presents a novel integration of olfactory data with vision modalities to enhance multimodal perception, introducing a new sensory dimension beyond vision, audio, and touch. Although traditional CAPTCHA systems rely on visual and audio signals to discriminate humans from bots, visuo-olfactory integration could inspire future multimodal challenge designs leveraging chemical sensing, although practical deployment is constrained by sensor availability.

From a defense perspective, See & Sniff demonstrates dense local alignment techniques and synthetic multimodal pairing strategies that may be generalized to other modalities or cross-modal verification tasks. The smell localization task also exemplifies fine-grained grounding of non-visual signals within natural scenes, potentially informing spatially grounded verification challenges. However, the field remains nascent with limitations in real-world dataset scale and hardware accessibility, so immediate CAPTCHA applications are exploratory rather than ready-made.

Cite

@article{arxiv2606_27307,
  title={ See & Sniff: Learning Visuo-Olfactory Representations },
  author={ Seongyu Kim and Seungwoo Lee and Hyeonggon Ryu and Joon Son Chung and Arda Senocak },
  journal={arXiv preprint arXiv:2606.27307},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.27307}
}

Read the full paper

• arXiv:2606.27307 (abstract)
• arXiv:2606.27307 (PDF)