VEGAS: Human-Aligned Video Caption Evaluation via Gaze

2026-07-09 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

VEGAS (Video caption Evaluation via GAze Score) is a novel, training-free metric designed to evaluate video captions by aligning them with individual viewer attention. While vision-language models often produce generic video descriptions, VEGAS leverages test-time gaze data to sample personalized, attention-aligned text. This cross-modal, information-theoretic metric quantifies how effectively a candidate caption matches a viewer's specific focus. To validate VEGAS, researchers curated a dataset comprising egocentric activities and instructional slides, complete with synchronized gaze data and reference annotations. Experiments demonstrated that captions selected using VEGAS via rejection sampling, without model retraining, significantly improved alignment with human focus. Furthermore, these VEGAS-selected captions enhanced downstream caption-to-video retrieval tasks, highlighting the practical benefits of integrating viewer attention during the inference phase of video captioning.

Key takeaway

For computer vision engineers developing video captioning systems, consider integrating viewer attention data during inference. VEGAS demonstrates that leveraging test-time gaze can significantly improve caption alignment with human focus and enhance downstream caption-to-video retrieval. This approach allows you to personalize generated descriptions without retraining large models, offering a practical path to more relevant and engaging video content.

Key insights

VEGAS uses test-time gaze data as a training-free, cross-modal metric to select video captions that align with individual viewer attention, improving retrieval.

Principles

• Viewer gaze data enhances caption relevance.
• Cross-modal metrics can personalize content.
• Attention alignment improves retrieval tasks.

Method

VEGAS quantifies caption-viewer focus match using an information-theoretic approach. It samples personalized captions via rejection sampling based on test-time gaze, without requiring model retraining.

In practice

• Integrate gaze data for personalized captions.
• Apply VEGAS for improved video retrieval.
• Use rejection sampling for attention alignment.

Topics

• Video Captioning
• Gaze Tracking
• Attention Alignment
• Cross-modal Metrics
• Vision-Language Models
• Information Theory

Best for: Research Scientist, AI Scientist, Computer Vision Engineer, NLP Engineer

Related on AIssential

• VEGAS: Human-Aligned Video Caption Evaluation via Gaze — VEGAS uses test-time gaze to personalize video captions, significantly improving alignment with human attention and enhancing retrieval without retraining.
• VEGAS: Human-Aligned Video Caption Evaluation via Gaze — VEGAS personalizes video captions by using test-time gaze to select descriptions aligned with individual viewer attention.
• Temporal and Cross-Modal Alignment for Enhanced Audiovisual Video Captioning — TCA-Captioner enhances audiovisual video captioning by achieving precise temporal and cross-modal alignment through an iterative refinement strategy.
• PercepCap: Video Captioner with Structured Spatio-Temporal Perception — PercepCap explicitly separates spatio-temporal perception from caption generation for improved video understanding.
• Hierarchical Multi-Modal Retrieval for Knowledge-Grounded News Image Captioning — Retrieval-augmented image captioning uses hierarchical multi-modal knowledge retrieval and LLMs to generate context-rich news image descriptions.
• Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction — The CRR framework disentangles text conditions for T2SV, resolving modal interference and bridging training-inference caption gaps.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.