Sidewalk Moments: Are Richer Representations Always More Human-Aligned? Evidence from City-Walk Videos

2026-07-23 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Human-Computer Interaction · Depth: Expert, quick

Summary

A study investigated whether richer visual representations consistently align better with human perceptions of urban engagement, using 61 first-person city-walk videos from YouTube, segmented into over 50,000 ten-second clips. Researchers analyzed four modalities: spatiotemporal video features, temporally averaged images (TAIs), audio embeddings, and text-based semantic descriptions. While Spearman correlation indicated video features offered the strongest continuous alignment, binary classification of high versus low engagement revealed TAIs consistently matched or surpassed full video across most classifiers. An independent Amazon Mechanical Turk study corroborated this, showing human participants identified engaging moments with comparable accuracy from TAIs and full video clips. Text performed substantially worse, and audio remained near chance. The findings suggest video features are superior for activity-driven scenes, whereas TAIs align better with human judgments in composition-driven scenes, challenging the notion that richer representations are always more human-aligned.

Key takeaway

For Computer Vision Engineers developing perceptual scoring models for urban engagement, you should re-evaluate the assumption that richer, full video representations are always superior. Your models might achieve comparable human alignment using temporally averaged images (TAIs), particularly for composition-driven scenes with stable spatial structure. This approach can offer a principled alternative to full video encoding, potentially reducing computational overhead without sacrificing accuracy in many contexts. Consider testing TAIs as a primary representation, especially when processing large volumes of city-walk data.

Key insights

Richer visual representations do not inherently guarantee greater human alignment for urban engagement tasks.

Principles

• Perceptually grounded temporal compression can be a principled alternative to full video encoding.
• Representation effectiveness varies by scene type: video for activity, TAIs for composition.

Method

Segmented 61 city-walk videos into 50,000 ten-second clips, analyzed four modalities (video, TAIs, audio, text) via Spearman correlation and binary classification, and validated with an Amazon Mechanical Turk study.

In practice

• Prioritize TAIs for urban engagement models in composition-driven scenes.
• Select full video features for activity-driven scenes with dynamic content.

Topics

• City-walk Videos
• Urban Engagement
• Visual Representations
• Temporally Averaged Images
• Human Alignment
• Perceptual Scoring Models

Best for: AI Scientist, Research Scientist, Computer Vision Engineer

Related on AIssential

• Improving Viewpoint-Invariance and Temporal Consistency for Action Detection — A two-stage approach using augmented virtual viewpoints and a multi-scale temporal encoder improves action detection's viewpoint-invariance and temporal consistency.
• AI doesn’t ‘see’ the way that you do, and that could be a problem when it categorizes objects and scenes — AI vision systems misclassify objects due to reliance on surface textures, not human-like shape and context.
• HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining — Carefully processed egocentric human video can surprisingly outperform real-robot data for embodied foundation model pretraining, offering a scalable alternative.
• VEGAS: Human-Aligned Video Caption Evaluation via Gaze — VEGAS personalizes video captions by using test-time gaze to select descriptions aligned with individual viewer attention.
• Making Foresight Actionable: Repurposing Representation Alignment in World Action Models — Aligning visual representations with action-grounded semantics improves robot manipulation policies in World Action Models.
• TROPHIES: Temporal Reconstruction of Places, Humans, and Cameras from Multi-view Videos — TROPHIES unifies human, scene, and camera reconstruction from multi-view videos into a globally consistent 4D space.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.