Revisiting Scene Graph Generation from the Perspective of Detector-Conditioned Reachability

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, extended

Summary

A new Dual-SGG method addresses the distinct predictive behaviors of detector-based and query-based Scene Graph Generation (SGG) approaches. Detector-based models struggle with triplets whose entities are semantically or spatially uncovered by object detections, a "detector constraint." Query-based models improve on these "unreachable" triplets but often drop performance on "reachable" ones. The Dual-SGG method integrates both reasoning mechanisms via a dual-query design (Top-Down Queries (TD-Qs) and Bottom-Up Queries (BU-Qs)) within a unified triplet decoder. This approach maintains strong performance on detector-reachable triplets while significantly improving performance on unreachable ones. Extensive experiments on Visual Genome, Open Images v6, and GQA-200 datasets demonstrate its effectiveness, achieving state-of-the-art results, for example, F@K of 23.6/27.4 and mR@K of 22.2/25.2 on Visual Genome with logit adjustment.

Key takeaway

For Machine Learning Engineers developing Scene Graph Generation systems, consider adopting the Dual-SGG architecture. This method effectively combines the strengths of detector-based and query-based approaches, ensuring robust triplet prediction even for entities challenging for traditional detectors. Your models will achieve superior overall performance on datasets like Visual Genome, Open Images v6, and GQA-200, reducing the need for complex post-hoc fusion strategies and improving high-level image understanding applications.

Key insights

Integrating detector-based and query-based SGG mechanisms via a dual-query design resolves their complementary prediction discrepancies.

Principles

• Detector-based SGG excels on "reachable" entities, struggles with "unreachable."
• Query-based SGG improves "unreachable" entities but may degrade "reachable."
• Jointly updating dual queries in a single decoder fuses complementary behaviors.

Method

Dual-SGG uses an object detector, an Entity Pair Selector (EPS) for Top-Down Queries (TD-Qs) from detected pairs, and Bottom-Up Queries (BU-Qs) initialized from the image center. Both query types are jointly updated in a single transformer triplet decoder.

In practice

• Apply Dual-SGG for robust scene graph generation across diverse entity reachability.
• Use center-biased initialization for bottom-up queries to explore wider regions.
• Employ self-attention masks to prevent information leakage between query types.

Topics

• Scene Graph Generation
• Object Detection
• Query-based Models
• Transformer Decoders
• Visual Genome
• Computer Vision

Code references

• runfeng-q/Revisiting-Scene-Graph-Generation-from-the-Perspective-of-Detector-Conditioned-Reachability

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Revisiting Scene Graph Generation from the Perspective of Detector-Conditioned Reachability — Detector-based and query-based SGG methods offer complementary predictive behaviors.
• Learning Where to Embed: Noise-Aware Positional Embedding for Query Retrieval in Small-Object Detection — Selectively embedding positional information in foreground regions improves small-object detection and reduces model complexity.
• DGSG-Mind: Dynamic 3D Gaussian Scene Graphs for Long-Term Scene Understanding and Grounding — DGSG-Mind uses hybrid 3D Gaussian scene graphs and an embodied reasoning agent for robust, dynamic, long-term scene understanding.
• Global-Local Monte Carlo Tree Search in Vision-Language Models for Text-to-3D Indoor Scene Generation — PRM-guided MCTS with hierarchical scene representation enables LVLMs to generate realistic 3D indoor scenes by correcting errors and balancing search.
• SToRe3D: Sparse Token Relevance in ViTs for Efficient Multi-View 3D Object Detection — Jointly sparsifying 2D image tokens and 3D object queries with relevance-aligned filtering significantly boosts multi-view 3D detection efficiency.
• Video Scene Graph Generation Using VLMs — VLMs enable open-vocabulary video scene graph generation, overcoming fixed ontology limitations for richer video understanding.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.