Why Vision Language Models Ignore What They See with Munawar Hayat - #758

2025-12-09 · Source: The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence) · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision & Image Processing, Natural Language Processing · Depth: Expert, extended

Summary

Munawar Hayat, a researcher at Qualcomm AI Research, discussed several papers presented at NeurIPS 2025 focusing on multimodal and generative AI, particularly for efficient on-device deployment. Key challenges addressed include object hallucination in Vision-Language Models (VLMs), where models often prioritize language priors over visual information. His team developed an attention-guided alignment method to improve visual grounding and introduced a novel generalized contrastive learning approach for complex, composed retrieval tasks, such as combined text and image queries, without increasing inference costs. Additionally, they tackled difficulties in generative models rendering multiple human subjects, creating the "MultiHuman Testbench" to measure and mitigate issues like identity leakage and attribute blending. Qualcomm AI Research aims to advance AI capabilities for perception, reasoning, and action across devices, with a strong presence at NeurIPS, including 17 papers and nine demos.

Key takeaway

For Computer Vision Engineers developing multimodal AI for on-device deployment, you should prioritize techniques that enforce strong visual grounding and prevent object hallucination. Consider adopting generalized contrastive learning for robust cross-modal retrieval and explore attention-masking strategies to improve identity preservation in multi-person image generation, ensuring efficient and accurate mobile AI experiences.

Key insights

VLMs often ignore visual input, relying on language priors, necessitating improved visual grounding and multimodal alignment.

Principles

• Vision models require explicit visual grounding.
• Multimodal retrieval benefits from generalized contrastive learning.
• Multi-person generation needs identity preservation.

Method

Attention-guided alignment injects visual tokens hierarchically with an auxiliary loss. Generalized contrastive learning reformulates loss using combinations of image, text, and fused embeddings. Multi-Human Testbench uses attention masks to prevent identity leakage.

In practice

• Use attention-guided alignment to reduce VLM hallucination.
• Apply generalized contrastive learning for complex cross-modal search.
• Employ attention masks for multi-person image generation.

Topics

• Multimodal AI
• Generative AI
• Vision-Language Models
• Contrastive Learning
• On-Device AI

Best for: Computer Vision Engineer, Research Scientist, AI Researcher, AI Scientist, Machine Learning Engineer

Related on AIssential

• Why Vision Language Models Ignore What They See [Munawar Hayat] - 758 — Current VLMs struggle with physics-based generation, visual grounding, and multi-person image fidelity, necessitating improved training and architectural approaches.
• Understanding Cross-Modal Contributions in Continual Vision-Language Models: A Theoretical Perspective — A new theoretical perspective clarifies cross-modal contributions in continual vision-language models, addressing catastrophic forgetting.
• International Conference on Learning Representations (ICLR) 2026 — Apple's ICLR 2026 contributions span LLM reasoning, efficient training, computer vision, and AI safety.
• Multilingual OCR-Aware Fine-Tuning and Prompt-Guided Chain-of-Thought Reasoning for Multimodal Large Language Models — OCR-aware data curation and fine-tuning significantly improve MLLM robustness for multilingual text in degraded images.
• Visual-Seeker: Towards Visual-Native Multimodal Agentic Search via Active Visual Reasoning — Visual-Seeker introduces active visual reasoning for multimodal search, dynamically using fine-grained visual evidence.
• The Cost of Language: Centroid Erasure Exposes and Exploits Modal Competition in Multimodal Language Models — Language representations in MLMs systematically overshadow vision, causing underperformance in visual tasks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence).