Sketch and Text Synergy: Fusing Structural Contours and Descriptive Attributes for Fine-Grained Image Retrieval

2026-04-21 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

Researchers from Xidian University and the First Aircraft Design Institute have developed the Sketch and Text Based Image Retrieval (STBIR) framework, designed to improve fine-grained image retrieval using both hand-drawn sketches and textual descriptions. This framework addresses the inherent modality gaps where sketches provide structural contours but lack color/texture, and text offers attributes but omits spatial details. STBIR integrates a curriculum learning-driven robustness enhancement module to handle varying query quality, a category-knowledge-based feature space optimization module to boost representational power, and a multi-stage cross-modal feature alignment mechanism to mitigate distribution mismatches. The team also curated the STBIR benchmark dataset, comprising STBIR-S, STBIR-C (single-category fine-grained retrieval for shoes and chairs), and STBIR-D (large-scale daily objects), to validate the framework. Extensive experiments demonstrate STBIR's superior performance over existing state-of-the-art methods across these datasets.

Key takeaway

For Computer Vision Engineers developing fine-grained image retrieval systems, the STBIR framework offers a robust approach to combine sketch and text inputs. You should consider implementing its multi-stage cross-modal feature alignment, starting with sketch mapping, then image refinement, and finally text integration, to overcome modality gaps and improve retrieval accuracy, especially for complex, detailed queries. This method can significantly enhance your system's discriminative power and robustness against low-quality inputs.

Key insights

Fusing sketches and text via a multi-stage alignment framework significantly enhances fine-grained image retrieval performance.

Principles

• Modality complementarity improves retrieval.
• Curriculum learning enhances model robustness.
• Category knowledge optimizes feature space.

Method

The STBIR framework uses a curriculum learning module for robustness, a category-knowledge module for feature optimization, and a multi-stage cross-modal alignment mechanism, with CLIP as the feature encoder.

In practice

• Prioritize sketch feature mapping in multimodal alignment.
• Fine-tune image encoder before text for visual affinity.
• Use Qwen for structured text description generation.

Topics

• Fine-grained Image Retrieval
• Sketch and Text Synergy
• Cross-modal Feature Alignment
• Curriculum Learning
• Feature Space Optimization

Best for: Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Sketch and Text Synergy: Fusing Structural Contours and Descriptive Attributes for Fine-Grained Image Retrieval — Fusing sketch and text modalities enhances fine-grained image retrieval by combining structural and descriptive cues.
• Exploring Multi-Modal Large Language Models and Two-Stage Fine-Tuning for Fashion Image Retrieval — Integrating LLaVA and two-stage fine-tuning significantly enhances fashion image retrieval by generating attribute-aware data and improving contrastive learning.
• Towards Resolving Optimization Conflicts Between Image- and Text-Based Person Re-Identification — Decoupling I2I and T2I ReID training with a two-stage pipeline resolves optimization conflicts for better shared representations.
• VietFashion: Benchmarking Sketch-Text Composed Image Retrieval for Cultural Outfits — Cultural garment retrieval requires benchmarks that capture subtle structural and semantic details.
• From Scenes to Elements: Multi-Granularity Evidence Retrieval for Verifiable Multimodal RAG — Element-level visual evidence retrieval significantly enhances multimodal RAG verifiability and performance.
• CLIP Architecture for Abdominal CT Image-Text Alignment and Zero-Shot Learning: Investigating Batch Composition and Data Scaling — Random sampling and alternating batching are more effective than explicit class balancing for 3D medical vision-language models.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.