Video2Reaction: Mapping Video to Audience Reaction Distribution in the Wild

2026-07-08 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Media & Entertainment · Depth: Expert, quick

Summary

The Video2Reaction dataset is introduced, a multimodal resource mapping over 10,000 short movie segments to distributions of induced audience emotions derived from social media. This dataset serves as both a training resource and a reliable benchmark for predicting audience reactions to video content. To enable continuous, cost-effective annotation, a two-stage multi-agent pipeline utilizing only open-source LLMs was developed, achieving 86% correctness under blind human verification. The research establishes the first benchmark for video-to-reaction-distribution prediction in the wild. It demonstrates that while pretrained foundation video models perform poorly in zero-shot settings, finetuning them transforms them into effective predictors for full reaction distributions and dominant responses. However, the task remains challenging, with the strongest method, LLaVA-Next, achieving only 77% Top-3 F1 in dominant reaction prediction.

Key takeaway

For Machine Learning Engineers developing content recommendation systems, you should consider integrating the Video2Reaction dataset to train and benchmark models for predicting audience emotional responses. While pretrained video models require finetuning, this resource offers a path to model collective audience reactions, improving content engagement metrics. You can also explore the described LLM-powered annotation pipeline for similar subjective data collection tasks.

Key insights

The Video2Reaction dataset and LLM-powered pipeline enable benchmarking and training for predicting audience emotion distributions from video content.

Principles

• Audience reactions are quantifiable distributions.
• LLMs can cost-effectively annotate subjective data.
• Finetuning improves foundation video models.

Method

A two-stage multi-agent pipeline using open-source LLMs performs cost-effective, continuous annotation of audience emotion distributions from video content, achieving 86% correctness.

In practice

• Use Video2Reaction for model training.
• Benchmark video models on reaction prediction.
• Explore LLM agents for subjective annotation.

Topics

• Video Analysis
• Audience Reaction Prediction
• Multimodal Datasets
• Large Language Models
• Emotion Recognition
• LLaVA-Next

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction — Staged multimodal fusion with independent encoder pre-training improves robustness for continuous emotion intensity prediction.
• Foresee-to-Ground: From Predictive Temporal Perception to Evidence-Driven Reasoning for Video Temporal Grounding — F2G improves video temporal grounding by decoupling event identification from precise boundary measurement using citable evidence.
• I created an LLM trained solely on Jeffrey Epsteins emails to see how messed up it becomes :) — Fine-tuning an LLM on a controversial, unannotated dataset can reveal unique behavioral shifts and spark technical debate.
• X+Slides: Benchmarking Audience-Conditioned Slide Generation — Benchmarking LLM-generated slides requires audience-specific utility and source grounding, not just completeness or visual quality.
• Can LLMs Think Like Consumers? Benchmarking Crowd-Level Reaction Reconstruction with ConsumerSimBench — LLMs struggle to reconstruct specific, socially grounded consumer reactions in public discourse, despite strong technical performance.
• An Empirical Study on Learning Latent Representations for Emotional Speech Synthesis — Integrating speaker embedding and prosody bottleneck into FastSpeech 2 enables effective emotional speech synthesis and style transfer.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.