VEGAS: Human-Aligned Video Caption Evaluation via Gaze
Summary
VEGAS (Video caption Evaluation via GAze Score) is a novel, training-free metric. It addresses the issue of vision-language models generating video captions that often fail to align with individual viewer attention. This cross-modal, information-theoretic metric quantifies how effectively a candidate caption matches a viewer's specific focus. It leverages test-time gaze data to sample personalized, attention-aligned text. To validate VEGAS, researchers curated a dataset comprising egocentric activities and instructional slides, complete with synchronized gaze and reference annotations. Using VEGAS, captions were selected via rejection sampling without requiring any model retraining. Experiments demonstrated that VEGAS-selected captions significantly better align with human focus. They also enhance downstream caption-to-video retrieval tasks, highlighting the practical benefits of integrating viewer attention during the inference phase.
Key takeaway
For Machine Learning Engineers developing video captioning or retrieval systems, if you are struggling with generic outputs, VEGAS offers a compelling solution. You should consider integrating test-time gaze data during inference to personalize captions. This approach requires no model retraining. It significantly improves caption alignment with human focus and enhances downstream retrieval tasks. This directly addresses user engagement and relevance challenges.
Key insights
VEGAS uses test-time gaze to personalize video captions, significantly improving alignment with human attention and enhancing retrieval without retraining.
Principles
- Viewer attention enhances caption relevance.
- Gaze data personalizes model outputs.
- Metrics can use real-time user focus.
Method
VEGAS is a training-free, cross-modal, information-theoretic metric. It quantifies candidate caption alignment with viewer gaze, then selects personalized captions via rejection sampling during inference.
In practice
- Personalize video captions with gaze.
- Enhance caption-to-video retrieval.
- Integrate viewer attention during inference.
Topics
- Video Captioning
- Gaze Tracking
- Vision-Language Models
- Personalization
- Information-Theoretic Metrics
- Rejection Sampling
Best for: Research Scientist, Computer Vision Engineer, AI Scientist, Machine Learning Engineer
Related on AIssential
See Counsel's argued verdicts on the open AI decisions leaders are weighing →
Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.