Structured-Condensed Prompt Tuning in Vision-Language Models for Fine-grained Image Recognition

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, extended

Summary

The paper introduces Structured-Condensed Prompt Tuning (SCPT), a novel method for fine-grained image recognition (FGIR) using Vision-Language Models (VLMs) like CLIP. Addressing the limitation of existing prompt tuning methods that treat class labels as isolated entities, SCPT enhances semantic structure modeling. It incorporates Semantic Relation Encoding (SRE) to explicitly model inter-class semantic topology and a Semantic Condensation loss (ScLoss) to suppress redundant supervision and extract discriminative components. Extensive experiments on 14 fine-grained benchmarks, including Dog Breed, Oxford Pets, and Stanford Cars, demonstrate SCPT's superior performance, achieving an average accuracy of 76.70% in 16-shot few-shot learning and a 71.15% harmonic mean in base-to-novel generalization, outperforming methods like TCP.

Key takeaway

For AI Scientists and Machine Learning Engineers developing fine-grained image recognition systems, SCPT offers a significant advancement. Your models can achieve superior performance in few-shot and base-to-novel generalization by adopting SCPT's approach to explicitly model inter-class semantic relationships and condense supervision signals. Consider integrating Semantic Relation Encoding and Semantic Condensation loss to improve discriminability and robustness, especially when working with visually similar categories.

Key insights

SCPT enhances VLM fine-grained recognition by explicitly modeling inter-class semantic relationships and condensing supervision signals in prompt tuning.

Principles

• Explicitly model inter-class semantic topology for fine-grained tasks.
• Suppress redundant supervision signals to extract discriminative components.
• Structured semantic modeling improves few-shot adaptation and generalization.

Method

SCPT uses Semantic Relation Encoding (SRE) via signed random projection to capture inter-class topology and Semantic Condensation loss (ScLoss) with SVD-based denoising and adaptive K selection to refine supervision.

In practice

• Apply SVD-based denoising to text embeddings for cleaner signals.
• Use signed random projection for compact semantic representation.
• Consider adaptive K selection for SVD truncation in prompt tuning.

Topics

• Prompt Tuning
• Vision-Language Models
• Fine-grained Image Recognition
• Semantic Relation Encoding
• Semantic Condensation Loss
• Few-shot Learning
• Base-to-Novel Generalization

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Finetuning and Bulk Labelling Images with Prodigy — Fine-tuning pre-trained models with minimal annotations significantly improves image embeddings for more effective bulk labeling.
• Linear Recurrent Unit with Semantic Modulation for Image Super-Resolution — A Linear Recurrent Unit (LRU) combined with a Semantic Modulating Unit (SMU) significantly enhances single-image super-resolution performance and efficiency.
• Improving Vision-Language-Action Model Fine-Tuning with Structured Stage and Keyframe Supervision — Structured supervision significantly enhances Vision-Language-Action model fine-tuning for robotic manipulation.
• Multi-step Large Language Model for Fine-Grained Feedback in Stepwise Linear Equation Solutions — Multi-step LLMs and hybrid architectures significantly improve fine-grained error classification in algebraic problem solving for educational feedback.
• IEEE Transactions on Neural Networks and Learning Systems, Volume 37, Issue 3, March 2026 — The issue highlights advancements across neural networks, computer vision, and machine learning applications.
• MINDS at SemEval-2026 Task 9: A Multi-Paradigm Approach to Cross-Lingual Polarization Detection — The study compares three NLP paradigms for cross-lingual polarization detection across 22 languages, finding varied strengths.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.