DeepGaze3.5-VL: Modeling Scanpaths via Autoregressive Token Prediction

2026-07-02 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, quick

Summary

DeepGaze3.5-VL introduces a novel approach to modeling human visual scanpaths by framing prediction as a discrete sequence modeling task. This method maps spatial coordinates into a text vocabulary, leveraging pretrained Vision-Language Models (VLMs) to absorb diverse factors like viewer identities and task-specific objectives through simple prompting. Unlike rigid, specialized architectures, DeepGaze3.5-VL offers flexible conditioning and easy extendability, integrating per-fixation attributes such as durations. The autoregressive alignment enables scalable, exact computation of per-fixation log-likelihoods, equivalent to Information Gain (IG). The model achieves a new state-of-the-art, reaching 2.18 bits of IG on MIT1003, a 46% improvement over DeepGaze III, even against baselines using identical high-capacity vision encoders. Beyond prediction, it serves as a generative framework for controlled in-silico behavioral interventions, recovering known oculomotor phenomena from data.

Key takeaway

For AI Scientists and Machine Learning Engineers developing models for human visual attention, DeepGaze3.5-VL demonstrates that framing scanpath prediction as an autoregressive token prediction task using Vision-Language Models offers superior flexibility and performance. You should consider this VLM-based approach to integrate diverse conditioning factors like viewer identity or task-specific objectives, moving beyond rigid, specialized architectures. This method also provides a powerful tool for in-silico behavioral simulations, enabling controlled experiments difficult to conduct in vivo.

Key insights

DeepGaze3.5-VL models visual scanpaths as autoregressive token prediction using Vision-Language Models.

Principles

• Mapping coordinates to text vocabulary enables VLM leverage.
• Autoregressive alignment allows exact log-likelihood computation.
• Generative frameworks can simulate behavioral interventions.

Method

Frame scanpath prediction as discrete sequence modeling by mapping coordinates to a text vocabulary. Leverage pretrained Vision-Language Models for autoregressive token prediction, integrating conditioning via prompting.

In practice

• Integrate task-specific instructions into scanpath prediction.
• Capture personalized visual biases using viewer identities.
• Perform in-silico simulations of oculomotor phenomena.

Topics

• Visual Attention
• Scanpath Prediction
• Vision-Language Models
• Autoregressive Models
• Cognitive Modeling
• DeepGaze3.5-VL
• In-silico Simulation

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Improving 3D Labeling in Self-Driving by Inferring Vehicle Information using Vision Language Models — VLMs can significantly enhance 3D vehicle auto-labeling in self-driving by inferring precise dimensions and VMMGR, even correcting human errors.
• MotionEnhancer: Leveraging Video Diffusion for Motion-Enhanced Vision-Language Models — Video Diffusion Model attention provides motion-calibrated priors to enhance Vision-Language Model motion understanding via attention alignment.
• Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors — Stream3D-VLM enables real-time 3D spatial understanding from video streams by integrating incremental geometry and efficient voxel compression.
• From Bounding Boxes to Visual Reasoning: An On-Policy Data Annotation Tool for Vision-Language Models — ScreenAnnotator unifies spatial, semantic, and structural data for VLM training using an on-policy, synthesis-driven annotation tool.
• TimeVista: Exploring and Exploiting Vision-Language Models as Judges for Time Series Forecasting — Vision-Language Models reliably judge time series forecasts, aligning significantly better with human preferences than traditional metrics.
• Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Models — Visual token reduction in VLMs should prioritize recoverable routing over irreversible pruning due to dynamic token importance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.