Analysis-by-Proxy: Localization Signals in VLMs Operating as Condition Encoders

2025-02-22 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision & Image Processing · Depth: Expert, extended

Summary

The "Analysis-by-Proxy" framework investigates why Vision-Language Models (VLMs) used as single-pass condition encoders in diffusion-based image editing pipelines often fail to localize edits accurately, despite standalone VLMs demonstrating strong localization capabilities. The study, focusing on Qwen-Image-Edit with its Qwen2.5-VL-7B backbone, found a 31.5% performance gap: standalone VLMs achieved 89.0% localization accuracy, while the full pipeline managed only 57.5%. This discrepancy arises because crucial spatial signals are not reliably propagated to the final layers typically used for conditioning. Instead, these signals reside in intermediate VLM representations, at locations that dynamically shift based on the input prompt. The framework uses a lightweight Q-Former proxy to uncover these hidden signals, demonstrating that recovering and integrating them significantly improves edit localization.

Key takeaway

For Machine Learning Engineers developing diffusion-based image editing pipelines, if you are struggling with localization accuracy in complex, multi-entity scenes, you should re-evaluate your VLM feature extraction strategy. Instead of relying on final-layer representations, dynamically identify and extract spatial signals from intermediate VLM layers. This targeted approach, potentially using a proxy model to generate explicit bounding box cues, can significantly improve edit precision by providing more accurate spatial conditioning to the Diffusion Transformer.

Key insights

VLMs as condition encoders lose localization accuracy because spatial signals are hidden in intermediate layers, not exposed by standard extraction.

Principles

• VLM spatial knowledge is optimized for autoregressive generation.
• Final VLM layers over-abstract crucial spatial cues.
• Localization signals peak in input-dependent intermediate layers.

Method

Analysis-by-Proxy trains a lightweight Q-Former proxy on VLM intermediate representations for an auxiliary localization task, using GIoU and L1 losses to predict bounding boxes.

In practice

• Use intermediate VLM layers for spatial conditioning.
• Dynamically select VLM layers based on input context.
• Overlay predicted bounding boxes for explicit conditioning.

Topics

• Vision-Language Models
• Image Editing
• Diffusion Models
• Localization
• Mechanistic Interpretability
• Condition Encoders

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Analysis-by-Proxy: Localization Signals in VLMs Operating as Condition Encoders — VLMs as condition encoders often hide critical localization signals in varying intermediate representations, not standard output layers.
• SpatialSV: Internalizing Interpretable 3D Spatial Awareness in MLLMs via Task-Oriented Visual Supervision — SpatialSV internalizes interpretable 3D spatial awareness in MLLMs by actively lifting 2D visual features into explicit 3D representations.
• Masked Diffusion Vision-Language Models for Temporal Action Localization — Masked diffusion vision-language models enable joint, iterative refinement of temporal boundaries and semantic content in untrimmed videos.
• You Typed a Few Words. The AI Painted a World. Here’s Exactly How. — Text-to-image AI converts language into visuals through a multi-stage pipeline involving text encoders, embeddings, and diffusion models.
• Segmentation before Answering: Pixel Grounding for MLLM Visual Reasoning — SegAnswer improves MLLM visual reasoning by using pixel-level segmentation masks instead of bounding boxes for precise region-of-interest grounding.
• [R] Can Vision-Language Models See Squares? Text-Recognition Mediates Spatial Reasoning Across Three Model Families — VLMs excel at text-based spatial tasks but struggle with non-textual visual patterns, indicating a strong implicit OCR pipeline.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.