U-shaped Multi-granularity Learning for Vision-Language Models

2026-07-16 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, quick

Summary

UPrompt, a novel U-shaped multi-granularity prompt learning framework, addresses the granularity dilemma in vision-language models (VLMs), particularly for dense prediction tasks. Traditional prompt learning struggles with global prompts lacking fine-grained awareness and local prompts ignoring contextual associations, limiting cross-task generalization. Inspired by U-Net's multi-level representation integration, UPrompt builds parallel multi-granularity representations in both visual and textual modalities. It employs coarse-to-fine cascaded enhancement to propagate global context for local detail refinement and fine-to-coarse hierarchical supervision to ensure semantic consistency across scales. Extensive experiments on 17 benchmarks demonstrate its effectiveness, outperforming MAMET and VPKE by 4.1 and 7.3 rSum on MSCOCO, and surpassing CoCoA-Mix by 5.09% in base-to-novel generalization, all while maintaining competitive performance with minimal overhead.

Key takeaway

For Machine Learning Engineers developing vision-language models for dense prediction tasks or seeking improved generalization, you should consider UPrompt to overcome the prompt granularity dilemma. Its U-shaped multi-granularity framework offers superior performance on benchmarks like MSCOCO and better base-to-novel generalization. Evaluate its integration for enhanced cross-task capabilities, potentially reducing computational costs while maintaining accuracy.

Key insights

UPrompt resolves VLM prompt granularity issues by integrating multi-level visual and textual representations, inspired by U-Net.

Principles

• Multi-granularity integration enhances VLM generalization.
• Symmetric encoder-decoder structures improve representation.
• Cross-level connections propagate context and refine details.

Method

UPrompt constructs parallel multi-granularity visual and textual representations. It uses coarse-to-fine cascaded enhancement and fine-to-coarse hierarchical supervision for semantic consistency.

In practice

• Apply UPrompt for dense prediction tasks.
• Improve base-to-novel generalization in VLMs.
• Reduce computational cost for competitive VLM performance.

Topics

• Vision-Language Models
• Prompt Learning
• Multi-granularity Learning
• U-Net Architecture
• Dense Prediction
• Cross-task Generalization
• MSCOCO Dataset

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Granulon: Awakening Pixel-Level Visual Encoders with Adaptive Multi-Granularity Semantics for MLLM — Granulon adaptively modulates visual granularity in MLLMs, enhancing both fine-grained detail and coarse-level semantic abstraction.
• UNIVID: Unified Vision-Language Model for Video Moderation — UNIVID unifies video moderation through policy-aware captions, enhancing interpretability and operational efficiency.
• Structured-Condensed Prompt Tuning in Vision-Language Models for Fine-grained Image Recognition — SCPT enhances VLM fine-grained recognition by explicitly modeling inter-class semantic relationships and condensing supervision signals in prompt tuning.
• Multi-Task Learning with Additive U-Net for Image Denoising and Classification — Additive skip fusion in U-Nets improves multi-task learning stability and performance by regularizing information flow.
• Beyond Attention Scores: SVD-Based Vision Token Pruning for Efficient Vision-Language Models — SVD-Prune uses global variance and leverage scores for training-free VLM token pruning, outperforming attention-based methods.
• The Professor: Multi-Teacher Unsupervised Prompt Distillation for Vision-Language Models — Multi-teacher prompt distillation significantly improves VLM student performance, particularly under domain shift, through complementary teacher supervision.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.