VEGAS: Human-Aligned Video Caption Evaluation via Gaze

2026-07-11 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

VEGAS (Video caption Evaluation via GAze Score) is a novel, training-free metric designed to align video captions with individual viewer attention. Developed by researchers at The University of Texas at Austin and AMD, VEGAS quantifies how well a candidate caption matches a viewer's focus by measuring pointwise conditional mutual information between caption tokens and non-attended visual regions, conditioned on attended regions. It operates by selecting captions with the lowest VEGAS score via rejection sampling from existing VLM outputs, without requiring model retraining. To validate VEGAS, a multimodal dataset was curated, combining egocentric videos from Aria Everyday Activities (AEA) and instructional slides from SlideVQA, totaling 632 visual samples and 2,981 human captions. Experiments showed VEGAS-selected captions significantly improved semantic alignment with human annotations on AEA, increasing mean SBERT similarity by +0.0856 and improving mAP by +2.48% at rank 5 in caption-to-video retrieval. Gains were less significant on SlideVQA, indicating VEGAS is most effective for concrete object disambiguation.

Key takeaway

For Machine Learning Engineers developing video captioning or retrieval systems, integrating VEGAS offers a practical, training-free method to personalize outputs. You should consider implementing VEGAS-based rejection sampling at inference time to align VLM-generated captions with individual user attention, especially for applications involving concrete object or action descriptions. This approach significantly improves semantic alignment and retrieval performance, enhancing user experience without requiring costly model retraining.

Key insights

VEGAS personalizes video captions by using test-time gaze to select descriptions aligned with individual viewer attention.

Principles

• Human video descriptions are diverse and gaze-tied.
• Gaze can proxy visual attention for personalized captioning.
• Lower VEGAS scores indicate better gaze-caption alignment.

Method

VEGAS quantifies caption-gaze alignment using pointwise conditional mutual information. It then employs rejection sampling to select the lowest-scoring caption from VLM-generated candidates, without VLM retraining.

In practice

• Enhance caption-to-video retrieval accuracy.
• Generate personalized video descriptions for users.
• Improve attention-aware human-computer interaction.

Topics

• Video Captioning
• Gaze Tracking
• Vision-Language Models
• Personalized AI
• Information Theory
• Rejection Sampling
• Video Retrieval

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• VEGAS: Human-Aligned Video Caption Evaluation via Gaze — VEGAS uses test-time gaze data as a training-free, cross-modal metric to select video captions that align with individual viewer attention, improving retrieval.
• Anomalous Frame Detection Using VLM-Based Description Comparison for Extracting Expert-Specific Actions and Contextual Decision-Making Scenes with Intra-Video Self-Similarity — A VLM-based method extracts expert actions and decision contexts from task videos by comparing visual descriptions.
• VEGAS: Human-Aligned Video Caption Evaluation via Gaze — VEGAS uses test-time gaze to personalize video captions, significantly improving alignment with human attention and enhancing retrieval without retraining.
• GAZE: Grounded Agentic Zero-shot Evaluation with Viewer-Level Tools and Literature Retrieval on Rare Brain MRI — Agentic VLMs with iterative tool use and retrieval enhance medical image interpretation, especially for rare conditions.
• STiTch: Semantic Transition and Transportation in Collaboration for Training-Free Zero-Shot Composed Image Retrieval — Training-free zero-shot CIR improves by refining LLM-generated captions and using set-to-set alignment for diverse compositions.
• Measuring the Effects of Visual Salience in Human and AI Descriptions with Image Editing — Visual salience, especially relational and perceptual, dictates entity prominence in image descriptions for both humans and AI.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.