Enhancing Gaze Reasoning in Vision Foundation Models for Gaze Following

2026-05-21 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

A novel training mechanism enhances gaze reasoning in Vision Foundation Models (VFMs) for gaze following, addressing a key limitation where VFMs improve scene understanding but not gaze reasoning. Existing VFM-based methods often rely on semantically salient objects, degrading performance when gaze targets lack salience. The proposed method introduces two components: a head-conditioned local LoRA, which enables localized adaptation to preserve scene token learning while improving head token learning for gaze reasoning, and an out-of-cone penalty, which injects gaze cues into head tokens while aligning them with scene tokens. Experiments on the GazeFollow and VAT datasets demonstrate leading performance, with particularly strong improvements observed when gaze targets are not semantically salient. This research, published on 2026-05-21, offers valuable insights for future gaze following advancements.

Key takeaway

For Computer Vision Engineers developing gaze following systems, if your models struggle with non-semantically salient gaze targets, consider integrating this novel training mechanism. Implementing a head-conditioned local LoRA and an out-of-cone penalty can significantly enhance gaze reasoning in your Vision Foundation Models, achieving top performance on challenging datasets like GazeFollow and VAT. This approach directly addresses a key VFM limitation.

Key insights

A novel training mechanism significantly improves Vision Foundation Models' gaze reasoning for gaze following, particularly with non-salient targets.

Principles

• VFMs require explicit gaze reasoning enhancement.
• Localized adaptation preserves scene token learning.
• Gaze cues improve head token alignment.

Method

The method employs a head-conditioned local LoRA for localized adaptation, preserving scene token learning while boosting head token learning. An out-of-cone penalty injects gaze cues into head tokens, aligning them with scene tokens.

In practice

• Improve gaze following for non-salient objects.
• Apply local LoRA for targeted VFM adaptation.
• Refine head token learning with gaze cues.

Topics

• Gaze Following
• Vision Foundation Models
• Gaze Reasoning
• LoRA Adaptation
• Computer Vision
• GazeFollow Dataset

Best for: Research Scientist, AI Scientist, Computer Vision Engineer, Machine Learning Engineer

Related on AIssential

• GAZE: Grounded Agentic Zero-shot Evaluation with Viewer-Level Tools and Literature Retrieval on Rare Brain MRI — Agentic VLMs with iterative tool use and retrieval enhance medical image interpretation, especially for rare conditions.
• Take a Peek: Efficient Encoder Adaptation for Few-Shot Semantic Segmentation via LoRA — Take a Peek (TaP) efficiently adapts FSS encoders to novel classes using Low-Rank Adaptation (LoRA) on support sets.
• LatRef-Diff: Latent and Reference-Guided Diffusion for Facial Attribute Editing and Style Manipulation — LatRef-Diff uses style codes and a novel modulation module for precise facial attribute and style manipulation.
• Reason, Then Re-reason: Cross-view Revisiting Improves Spatial Reasoning — Revisiting spatial hypotheses with synthesized complementary views significantly enhances MLLM reasoning from egocentric videos.
• Lazy or Efficient? Towards Accessible Eye-Tracking Event Detection Using LLMs — An LLM-driven pipeline automates eye-tracking event detection, making complex analysis accessible without specialized programming.
• On Asymmetric Optimization of Reasoning and Perception in Vision-Language Model Post-Training — VLM post-training creates a perception-reasoning asymmetry due to token imbalance (SFT) or reward coupling (RL), hindering end-to-end performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.