Dual-Correlation Hypergraph Network for Unaligned RGBT Video Object Detection and A Large-scale Benchmark

2026-07-09 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

A Dual-Correlation Hypergraph Network (DHNet) is proposed to enhance RGB-Thermal (RGBT) Video Object Detection (VOD) by addressing common spatial misalignment between RGBT image pairs. DHNet explicitly models temporal correlation across consecutive frames and spatial correlation from cross-modal features to capture high-dimensional complementary information. It integrates a Patch-based Spatial Alignment Module (PSAM) for local region feature alignment and a Dual Hypergraph Fusion Module (DHFM) that constructs separate temporal and multimodal hypergraphs for object discriminability. Complementing this, a new large-scale RGBT VOD benchmark, DVT-VOD1000, has been constructed, featuring 1,000 video sequences and 103,464 RGBT image pairs across diverse scenarios like night, rain, and snow. Experiments confirm DHNet achieves state-of-the-art detection accuracy on both VT-VOD50 and DVT-VOD1000. The dataset and source code will be publicly available.

Key takeaway

For Computer Vision Engineers developing robust object detection systems in challenging conditions, DHNet offers a significant advancement in RGBT VOD by effectively mitigating spatial misalignment. You should consider exploring its dual-correlation hypergraph network approach for enhanced object discriminability. Furthermore, the new DVT-VOD1000 benchmark provides a critical, large-scale resource for training and rigorously evaluating your RGBT VOD models across diverse, real-world scenarios, improving model generalization and performance.

Key insights

DHNet and DVT-VOD1000 advance RGBT VOD by resolving spatial misalignment and data scarcity through dual-correlation hypergraphs and a large-scale benchmark.

Principles

• Model temporal and spatial correlations for RGBT VOD.
• Hypergraph networks capture high-dimensional information.
• Diverse, large-scale benchmarks drive VOD progress.

Method

DHNet employs a Patch-based Spatial Alignment Module (PSAM) for local multimodal feature alignment. Subsequently, a Dual Hypergraph Fusion Module (DHFM) constructs temporal and multimodal hypergraphs, enhancing object discriminability via dual-correlation learning.

In practice

• Employ RGBT VOD for robust detection in adverse conditions.
• Integrate hypergraph networks for multimodal data fusion.
• Train RGBT VOD models using the DVT-VOD1000 dataset.

Topics

• RGBT Video Object Detection
• Hypergraph Networks
• Multimodal Fusion
• Spatial Alignment
• DVT-VOD1000
• Computer Vision

Best for: Research Scientist, AI Scientist, Computer Vision Engineer

Related on AIssential

• DRGBT-1K: A Large-scale High-quality Benchmark for Dynamic RGBT Tracking — DRGBT-1K provides a robust benchmark for evaluating dynamic RGBT tracking under real-world modality and platform variations.
• H2VLR: Heterogeneous Hypergraph Vision-Language Reasoning for Few-Shot Anomaly Detection — H2VLR uses a heterogeneous hypergraph for high-order visual-semantic reasoning in few-shot anomaly detection.
• Multi-Modal Hyper-Graph Fusion for Low-Light Crowd Counting — Multi-modal hyper-graph fusion significantly improves low-light crowd counting by integrating RGB, depth, and edge cues.
• 🔥BoxerNet: SOTA 2D->3D BBs🔥 👉Boxer by META: transformer-based network to lift 2D BB... — BoxerNet lifts 2D bounding box proposals to globally consistent 3D bounding boxes using transformers and multi-view fusion.
• HST-HGN: Heterogeneous Spatial-Temporal Hypergraph Networks with Bidirectional State Space Models for Global Fatigue Assessment — HST-HGN uses hypergraphs and Bi-Mamba for efficient, accurate driver fatigue detection by modeling high-order spatial synergies and long-range temporal dependencies.
• SWNet: A Cross-Spectral Network for Camouflaged Weed Detection — Cross-spectral fusion and edge refinement are crucial for accurate camouflaged weed detection in agriculture.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.