C-GAP: Class-Aware and Online Prompting Improves Vision-Language Models on Imbalanced Classes

2026-07-10 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

C-GAP (Caption-Guided Augmentation and Prompting) is a new detector-agnostic, annotation-free framework designed to improve vision-language models' detection of rare object classes in safety-critical perception systems. This framework addresses class imbalance by iteratively refining natural language prompts fed to frozen open-vocabulary detectors, rather than retraining models or collecting new annotations. C-GAP operates in two phases: first, it establishes a composite caption baseline by combining per-image scene descriptions with class-quantity context, which significantly outperforms single-context prompts. Second, an LLM iteratively refines individual image captions, triaging trials based on minority-class AP@0.5 against a dynamic threshold, with refinement stopping upon sufficient gain. Experiments demonstrate C-GAP improves minority-class average precision by up to 53% over baselines, achieving an ~81% relative improvement on COCO's minority-class AP@0.5 (from 17.69 to 32.09) compared to the composite baseline.

Key takeaway

For Machine Learning Engineers developing safety-critical perception systems, C-GAP offers a powerful approach to enhance rare object detection without model retraining. You should consider implementing C-GAP's two-phase prompt refinement, starting with composite captions and leveraging an LLM for iterative optimization. This method significantly improves minority-class average precision, allowing you to boost performance on imbalanced datasets efficiently and cost-effectively.

Key insights

Iteratively refining language prompts with an LLM, using composite captions, significantly boosts minority-class detection in frozen vision-language models.

Principles

• Prompt quality is a first-class lever for detection.
• Composite captions provide critical foundation for refinement.
• Iterative prompt refinement improves minority class AP.

Method

C-GAP establishes a composite caption baseline, then an LLM iteratively refines image captions. Trials are triaged by minority-class AP@0.5 against a dynamic threshold, terminating early on sufficient gain.

In practice

• Apply composite captions for open-vocabulary detection.
• Use LLMs for iterative prompt refinement.
• Improve rare object detection without model retraining.

Topics

• Vision-Language Models
• Class Imbalance
• Open-Vocabulary Detection
• Prompt Engineering
• Large Language Models
• Safety-Critical Systems
• Object Detection

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• NCL-UoR at SemEval-2026 Task 5: Embedding-Based Methods, Fine-Tuning, and LLMs for Word Sense Plausibility Rating — Structured prompting with explicit rules significantly enhances word sense plausibility rating, outweighing model scale.
• CPS4: Class Prompt driven Semi-Supervised Spine Segmentation with Class-specific Consistency Constraint — CPS4 enhances semi-supervised spine segmentation by using class prompts with explicit consistency constraints in VLMs.
• Addressing Data Scarcity in Bangla Fake News Detection: An LLM-Based Dataset Augmentation Approach — LLM-based data augmentation significantly improves fake news detection in low-resource languages by generating diverse synthetic data.
• Structured Data Retrieval with Sparrow using OCR and Vision LLM [Improved Accuracy] — Combining OCR with Vision LLMs via structured prompts improves large table data extraction accuracy and reduces hallucinations.
• Rethinking Cross-lingual Gaps from a Statistical Viewpoint — LLM cross-lingual gaps stem from response variance, not knowledge barriers, and are reducible via variance control.
• BEiTScore: Reference-free Image Captioning Evaluation with an Efficient Cross-Encoder Model — BEiTScore offers an efficient, reference-free image captioning evaluation metric, outperforming LLMs on complex tasks.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.