LongEgoRefer: A Benchmark for Long-Form Egocentric Video Referring Expression Comprehension

2026-07-02 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

LongEgoRefer is a new benchmark designed to address limitations in existing egocentric Video Referring Expression Comprehension (Video REC) datasets, which primarily use short video clips. Constructed from long-form videos within the Ego4D dataset, LongEgoRefer features 1,498 referring expressions with an average video duration of 45 minutes. This benchmark introduces a demanding spatio-temporal grounding problem due to its extreme target sparsity, detailed linguistic descriptions, and complex human-object interactions embedded in extended egocentric narratives. It requires models to identify both the temporal occurrence of an event and the spatial location of a referred object within long video sequences. Evaluations show that current advanced baselines and state-of-the-art Video REC models struggle significantly on LongEgoRefer, underscoring the inherent difficulty of long-form egocentric spatio-temporal grounding and the need for more robust video understanding models.

Key takeaway

For computer vision engineers developing Video Referring Expression Comprehension models, recognize that current approaches struggle significantly with long-form egocentric video. Your model development should prioritize robustness against sparse object occurrences and complex activity transitions found in untrimmed, extended recordings. Focus on spatio-temporal grounding solutions that can effectively identify both event timing and object location across 45-minute video sequences to meet real-world demands.

Key insights

LongEgoRefer challenges existing Video REC models by introducing a benchmark for long-form egocentric videos with sparse object occurrences.

Principles

• Long-form egocentric video presents unique grounding challenges.
• Sparse object occurrences complicate spatio-temporal localization.
• Real-world egocentric recordings are untrimmed and long-form.

In practice

• Evaluate Video REC models on long-form, sparse datasets.
• Integrate vision-language models for spatio-temporal grounding.
• Focus model development on untrimmed egocentric narratives.

Topics

• Egocentric Video
• Video Referring Expression Comprehension
• Long-Form Video Analysis
• Spatio-Temporal Grounding
• Ego4D Dataset
• Vision-Language Models

Best for: Research Scientist, AI Scientist, Computer Vision Engineer

Related on AIssential

• EgoEverything: A Benchmark for Human Behavior Inspired Long Context Egocentric Video Understanding in AR Environment — EgoEverything uses human attention signals to create a realistic benchmark for egocentric video understanding in AR.
• Flat-Pack Bench: Evaluating Spatio-Temporal Understanding in Large Vision-Language Models through Furniture Assembly — Current LVLMs significantly struggle with fine-grained spatio-temporal reasoning and object interaction in complex video scenarios.
• Objects Before Words: Object-First Inductive Biases for Grounding Language in Child-View Video — Object-first inductive biases improve language grounding in noisy child-view video by tracking objects across utterances.
• Hand Trajectory Fusion for Egocentric Natural Language Query Grounding — Integrating hand trajectory data significantly improves egocentric video NLQ grounding for interaction-focused queries.
• VideoDetective: Clue Hunting via both Extrinsic Query and Intrinsic Relevance for Long Video Understanding — Integrating extrinsic query relevance with intrinsic video structure enhances long video understanding for MLLMs.
• Reason, Then Re-reason: Cross-view Revisiting Improves Spatial Reasoning — Revisiting spatial hypotheses with synthesized complementary views significantly enhances MLLM reasoning from egocentric videos.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.