Language-Instructed Vision Embeddings for Controllable and Generalizable Perception

2026-06-17 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Language-Instructed Vision Embeddings (LIVE) introduces a new paradigm for vision foundation models, moving beyond static feature extraction. Instead of feeding visual features into large language models, LIVE uses language to dynamically guide the vision encoder itself. This method produces task-centric embeddings at inference time, eliminating the need for task-specific retraining. By focusing on contextually relevant aspects of input, LIVE yields more controllable and generalizable representations. Empirically, LIVE reduces visual hallucinations by 34 points on MMVP, outperforms vision-language models with significantly more parameters on visual question answering, and demonstrates generalization to unseen instructions and tasks, paving the way for adaptive, instruction-driven visual intelligence.

Key takeaway

For Machine Learning Engineers developing vision systems, LIVE offers a compelling alternative to traditional static feature extractors. You can achieve more controllable and generalizable perception without extensive task-specific retraining, significantly streamlining development. Consider integrating language-guided vision encoders to reduce visual hallucinations and improve performance on visual question answering, especially when adapting to novel instructions or tasks.

Key insights

Language dynamically guides vision encoders to produce task-centric embeddings, removing the need for task-specific retraining.

Principles

• Language can dynamically guide vision encoders.
• Task-centric embeddings enhance control and generalization.
• Inference-time guidance removes retraining needs.

Method

LIVE leverages language as high-level guidance to produce task-centric embeddings directly at inference time, enabling the encoder to focus on contextually relevant input aspects.

In practice

• Reduces visual hallucinations by 34 points on MMVP.
• Outperforms larger VLM on visual question answering.
• Generalizes to unseen instructions and tasks.

Topics

• Language-Instructed Vision Embeddings
• Vision Foundation Models
• Visual Question Answering
• Visual Hallucinations
• Generalizable Perception
• Dynamic Vision Guidance

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Diagnosing Visual Ignorance in Vision-Language Models — Vision-Language Models frequently prioritize language priors over visual evidence due to internal routing failures and flawed benchmark evaluations.
• Perceptual Flow Network for Visually Grounded Reasoning — PFlowNet improves LVLM visual reasoning by decoupling perception from reasoning and using variational reinforcement learning.
• Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs — Visual latents in MLLMs can be "silenced" by training objectives, but their reasoning capacity can be unleashed at inference time.
• Can Local Vision-Language Models improve Activity Recognition over Vision Transformers? -- Case Study on Newborn Resuscitation — Fine-tuned local VLMs with LoRA significantly outperform Vision Transformers for newborn resuscitation activity recognition.
• Granulon: Awakening Pixel-Level Visual Encoders with Adaptive Multi-Granularity Semantics for MLLM — Granulon adaptively modulates visual granularity in MLLMs, enhancing both fine-grained detail and coarse-level semantic abstraction.
• Retrieve, Don't Retrain: Extending Vision Language Action Models to New Tasks at Test Time — Adapting Vision-Language-Action models to new tasks can be achieved via demonstration retrieval, not costly per-task fine-tuning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.