Video Scene Graph Generation Using VLMs

2026-07-05 · Source: Naturallanguageprocessing on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Data Science & Analytics · Depth: Intermediate, medium

Summary

Video Scene Graph Generation (SGG) using Vision Language Models (VLMs) offers a novel approach to transform video content into structured, machine-readable descriptions. This method overcomes the limitations of traditional SGG, which relies on fixed vocabularies, by leveraging VLMs like Qwen2.5-VL to generate open-vocabulary (subject, predicate, object) triples directly from video frames. The process involves selecting keyframes using a histogram-based approach (e.g., max_frames=8, min_gap_sec=1, hist_threshold=0.35), then prompting the VLM to extract salient entities and interactions. The VLM, configured with max_new_tokens=512, outputs these relationships in a JSON schema, which is then parsed and used to build a NetworkX graph. This structured representation enables programmatic reasoning over video content, facilitating advanced applications like semantic video search and complex visual question answering.

Key takeaway

For AI Engineers developing video analysis solutions, this VLM-based scene graph generation method offers a powerful upgrade over traditional fixed-vocabulary approaches. You can now extract nuanced, open-ended relationships from video, enabling more sophisticated semantic search and multimodal reasoning capabilities. Consider integrating Qwen2.5-VL or similar VLMs to build dynamic, queryable representations of complex video interactions, moving beyond simple captioning.

Key insights

VLMs enable open-vocabulary video scene graph generation, overcoming fixed ontology limitations for richer video understanding.

Principles

• Scene graphs allow programmatic reasoning over video.
• Fixed vocabularies limit scene graph expressiveness.
• Keyframe selection is crucial for VLM efficiency.

Method

Select keyframes via histogram comparison, prompt a VLM (e.g., Qwen2.5-VL) for (subject, predicate, object) triples, then parse JSON and build a NetworkX graph.

In practice

• Implement semantic video search queries.
• Enhance visual question answering systems.
• Analyze complex spatial interactions.

Topics

• Video Scene Graph Generation
• Vision Language Models
• Qwen2.5-VL
• Keyframe Selection
• Semantic Video Search
• Visual Question Answering

Best for: Machine Learning Engineer, AI Engineer, AI Scientist

Related on AIssential

• Planning with the Views via Scene Self-Exploration — Self-exploration and view graph distillation enable VLMs to plan multi-turn camera movements in 3D environments.
• Grounding Qwen3-VL Detection with SAM2 — Combining Qwen3-VL's detection with SAM2's segmentation enables robust, natural language-driven object grounding.
• VGenST-Bench: A Benchmark for Spatio-Temporal Reasoning via Active Video Synthesis — VGenST-Bench shifts spatio-temporal reasoning evaluation from passive curation to active, controlled video synthesis for MLLMs.
• Occ-VLM: Occupancy Grounded Vision Language Model for Indoor Scene Understanding — Occ-VLM unifies 2D semantics and 3D geometry for scene understanding using only RGB images and occupancy as a spatial prior.
• ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding — Code-guided synthetic data generation significantly improves VLM chart understanding and reasoning capabilities.
• MotionEnhancer: Leveraging Video Diffusion for Motion-Enhanced Vision-Language Models — Video Diffusion Model attention provides motion-calibrated priors to enhance Vision-Language Model motion understanding via attention alignment.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Naturallanguageprocessing on Medium.