DINOv2 way worse than SigLIP in k-NN. Is this expected? [R]

2026-07-08 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Advanced, short

Summary

A user conducting a bachelor thesis on fine-grained car classification observed a significant performance disparity between DINOv2 and SigLIP when using a frozen encoder with weighted k-NN. On a small dataset (175 train / 132 test), SigLIP achieved 91% accuracy, while DINOv2 only reached 41%, a 50-point difference. This gap persisted even after checking for cosine vs. Euclidean similarity with L2-normalized embeddings. Explanations suggest DINOv2, a self-supervised semantic model, struggles with fine-grained distinctions due to augmentations creating invariances and lacking a contrastive component to separate similar objects. In contrast, SigLIP's contrastive training with language labels likely provides crucial fine-grained signal, potentially encoding features like color. An update noted that DINOv2 and CLIP can achieve up to 90% accuracy when used as backbones with a classification layer, such as an SVM head. DINOv3 is suggested as a superior alternative for retrieval, addressing DINOv2's patch embedding issues.

Key takeaway

Machine Learning Engineers choosing a vision backbone for fine-grained tasks should note that frozen self-supervised models like DINOv2 can underperform contrastive models such as SigLIP. If your application demands fine-grained distinctions, consider using DINOv2 or CLIP as a backbone. Pair it with a trained classification head, like an SVM, to achieve higher accuracy. Alternatively, evaluate DINOv3; it is noted to offer superior performance for retrieval.

Key insights

Self-supervised vision models like DINOv2 may underperform contrastive models like SigLIP in fine-grained k-NN retrieval without a trained head.

Principles

• Self-supervised models may lack fine-grained signal.
• Contrastive training with labels enhances distinction.
• DINOv3 offers improved patch embeddings.

Method

Use a frozen encoder for embeddings with weighted k-NN, or a frozen backbone (DINOv2/CLIP) with a classification layer (e.g., SVM) for better fine-grained results.

In practice

• Evaluate DINOv3 for retrieval tasks.
• Use DINOv2/CLIP with a classification head.
• Consider color encoding for car classification.

Topics

• Fine-grained Classification
• DINOv2
• SigLIP
• Contrastive Learning
• Self-supervised Learning
• DINOv3

Best for: AI Engineer, Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Student

Related on AIssential

• Technical Report for ICRA 2026 GOOSE 2D Fine-Grained Semantic Segmentation Challenge: Leveraging DINOv3 for Robust Outdoor Scene Understanding in Field Robotics — Combining DINOv3, Mask2Former, and test-time augmentation significantly boosts fine-grained semantic segmentation performance in challenging outdoor environments.
• SigLIP-HD by Fine-to-Coarse Supervision — SigLIP-HD enhances MLLM visual perception by using fine-to-coarse supervision, enabling high-quality visual tokens without increased computational cost or resolution.
• Rethinking Multi-Branch and Cross-Backbone Fusion for Vehicle Re-Identification in the Foundation-Model Era — Multi-branch and cross-backbone fusion for vehicle Re-ID offers no significant gains over a single, well-tuned foundation model.
• NVIDIA AI releases C-RADIOv4 vision backbone unifying SigLIP2, DINOv3, SAM3 for classification, dense prediction, segmentation workloads at scale — C-RADIOv4 unifies multiple vision models into a single backbone for diverse, high-performance computer vision tasks.
• Multi-Scale ViT Inference with Habitat-Fit Priors and kNN Retrieval for Multi-Species Plant Identification — Multi-scale ViT inference combined with habitat-fit priors and kNN retrieval significantly improves multi-species plant identification.
• DINOde: Continuous Vision-Text Alignment for Open-Vocabulary Semantic Segmentation — DINOde uses an ODE-based framework to continuously align CLIP text embeddings with DINO visual representations for state-of-the-art open-vocabulary semantic segmentation.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.