Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, extended

Summary

Scene Graph Thinking (SaGe) is a novel paradigm addressing Multimodal Large Language Models' (MLLMs) limitations in structured visual reasoning and fine-grained perception. SaGe introduces explicit scene-graph representations, starting with an automated data engine that converts flat image–text corpora into hierarchical scene graphs, comprising entities as nodes and visual relations as edges. This process generates 120K high-quality training data by sampling reasoning traces. A two-stage graph-aligned post-training approach follows, involving supervised fine-tuning to internalize structured reasoning and reinforcement fine-tuning with node-as-proxy graph rewards for efficient graph exploration. SaGe achieves significant improvements across eight multimodal benchmarks, with SaGe-3B outperforming GPT-4o and SaGe-7B surpassing Qwen2.5-VL-32B on tasks like VStarBench, HRBench-4K, and CVBench-2D/3D.

Key takeaway

For AI Scientists and Machine Learning Engineers aiming to enhance MLLM performance on fine-grained visual reasoning, traditional models often miss critical structured relationships. You should consider integrating explicit scene-graph representations and a two-stage graph-aligned training paradigm, like SaGe's, into your development. This approach, leveraging automated scene graph construction and reinforcement learning with node-as-proxy rewards, can significantly improve your models' perception and reasoning capabilities across complex visual benchmarks.

Key insights

SaGe enhances MLLMs' visual reasoning by integrating explicit scene-graph representations and a two-stage graph-aligned training.

Principles

• Structured visual data improves MLLM reasoning.
• Hierarchical scene graphs enable fine-grained perception.
• Node-as-proxy rewards guide efficient graph exploration.

Method

Automated engine converts image-text to scene graphs. Sample 120K data. Apply two-stage training: SFT for structured reasoning, then GRPO-based RFT with node-grounded and node-relevance rewards.

In practice

• Construct hierarchical scene graphs from images.
• Use node-articulated CoT for MLLM training.
• Implement node-as-proxy rewards for RL fine-tuning.

Topics

• Multimodal Large Language Models
• Scene Graph Representations
• Visual Reasoning
• Reinforcement Learning
• Fine-grained Perception
• Automated Data Generation

Code references

• zwyang6/SaGe

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• ISO-Bench: Benchmarking Multimodal Causal Reasoning in Visual–Language Models through Procedural Plans — Multimodal models struggle with causal reasoning, achieving only 0.57 F1 on ISO-Bench, far below human performance.
• Faithful-MR1: Faithful Multimodal Reasoning via Anchoring and Reinforcing Visual Attention — Faithful-MR1 improves MLLM reasoning by directly anchoring visual attention and reinforcing its causal use.
• Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs — Visual latents in MLLMs can be "silenced" by training objectives, but their reasoning capacity can be unleashed at inference time.
• Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphs — Explicit spatio-temporal scene graphs and graph-based models enable structured, interpretable reasoning about human-environment interactions in videos.
• Video Scene Graph Generation Using VLMs — VLMs enable open-vocabulary video scene graph generation, overcoming fixed ontology limitations for richer video understanding.
• Segmentation before Answering: Pixel Grounding for MLLM Visual Reasoning — Pixel-level segmentation significantly enhances MLLM visual reasoning by precisely isolating regions of interest, outperforming bounding box methods.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.