Exploring Multi-Modal Large Language Models and Two-Stage Fine-Tuning for Fashion Image Retrieval

2026-06-18 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision & Pattern Recognition · Depth: Expert, quick

Summary

A novel framework addresses limitations in composed image retrieval for fashion, a task requiring understanding subtle attribute variations. The proposed system integrates a multi-modal large language model (LLaVA) to generate attribute-aware triplets, overcoming scarce annotated data and simplistic negative sampling. It also introduces a two-stage fine-tuning strategy to enhance contrastive learning. The framework leverages pretrained vision-language models, specifically CLIP-ViT/B32, for generating and concatenating sentence-level prompts with relative captions and for scaling negatives using static representations. Experimental results, published on 2026-06-18, demonstrate enhanced compositional reasoning and improved fine-grained retrieval behavior, validating its potential for fashion retrieval.

Key takeaway

For Computer Vision Engineers or Machine Learning Engineers developing fashion image retrieval systems, this research offers a clear path to overcome data scarcity and improve fine-grained understanding. You should consider integrating multi-modal LLMs like LLaVA for generating attribute-aware training data and implementing a two-stage fine-tuning approach to enhance your contrastive learning models. This strategy can significantly improve compositional reasoning and retrieval accuracy in your applications.

Key insights

Integrating LLaVA and two-stage fine-tuning significantly enhances fashion image retrieval by generating attribute-aware data and improving contrastive learning.

Principles

• Attribute-aware triplet generation improves fine-grained retrieval.
• Two-stage fine-tuning enhances contrastive learning performance.
• Pretrained VLM features can scale negative sampling effectively.

Method

The framework integrates LLaVA for attribute-aware triplet generation and employs a two-stage fine-tuning strategy. It uses CLIP-ViT/B32 to create sentence-level prompts and scale negative samples via static representations.

In practice

• Utilize MLLMs like LLaVA for synthetic data generation.
• Implement multi-stage fine-tuning for complex visual tasks.
• Employ pretrained VLMs for efficient negative sampling.

Topics

• Multi-modal LLMs
• Fashion Image Retrieval
• Fine-tuning
• Contrastive Learning
• LLaVA
• CLIP-ViT/B32
• Composed Image Retrieval

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Hierarchical Multi-Modal Retrieval for Knowledge-Grounded News Image Captioning — Retrieval-augmented image captioning uses hierarchical multi-modal knowledge retrieval and LLMs to generate context-rich news image descriptions.
• Sketch and Text Synergy: Fusing Structural Contours and Descriptive Attributes for Fine-Grained Image Retrieval — Fusing sketch and text modalities enhances fine-grained image retrieval by combining structural and descriptive cues.
• VietFashion: Benchmarking Sketch-Text Composed Image Retrieval for Cultural Outfits — Cultural garment retrieval requires benchmarks that capture subtle structural and semantic details.
• ELVA: Exploring Ranking-Driven Universal Multimodal Retrieval — ELVA mitigates "grain blindness" in multimodal retrieval by ranking negative samples based on similarity, not just binary classification.
• From Scenes to Elements: Multi-Granularity Evidence Retrieval for Verifiable Multimodal RAG — Element-level visual evidence retrieval significantly enhances multimodal RAG verifiability and performance.
• I created an LLM trained solely on Jeffrey Epsteins emails to see how messed up it becomes :) — Fine-tuning an LLM on a controversial, unannotated dataset can reveal unique behavioral shifts and spark technical debate.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.