NERVE: A Neuromorphic Vision and Radar Ensemble for Multi-Sensor Fusion Research

2026-05-19 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Data Science & Analytics · Depth: Expert, extended

Summary

NERVE (Neuromorphic Vision and Radar Ensemble) is a new multi-sensor dataset designed for neuromorphic perception research, comprising 257 minutes of synchronized recordings from five sensors: two Dynamic Vision Sensors (DVS), an RGB-D camera, and two Radar units (24GHz and 77GHz). Captured over 12 days in office environments, the dataset totals 600 GB of uncompressed, temporally aligned data, featuring approximately 914,000 frames and 9.6 million COCO-formatted annotations across 16 object categories. The dataset includes an open-source software toolkit for preprocessing and format conversion. Baseline experiments on a DVS+Radar subset for human detection and distance estimation show that combining DVS with 77GHz Radar consistently improves detection, with recurrent models achieving up to 47.5% mAP and mean absolute Radar distance errors below 1.8 m against LiDAR ground truth.

Key takeaway

For research scientists developing perception systems for smart spaces, NERVE offers a unique benchmark for DVS-Radar fusion. You should explore recurrent neural network architectures, such as RVT-Tiny, to fully exploit the temporal dynamics of event-based and radar data, as these models significantly outperform feed-forward approaches. Consider the trade-off between high-resolution DVS for detection accuracy and lower-resolution options for potential power savings in IoT deployments.

Key insights

NERVE is a multi-sensor dataset bridging neuromorphic vision and radar for robust perception research.

Principles

• Multi-sensor fusion overcomes individual sensor limitations.
• Temporal context is critical for motion-dependent sensing.
• Higher spatial resolution improves detection performance.

Method

The dataset uses an automatic labeling pipeline with YOLOv8x on RGB frames, then maps annotations to DVS coordinates via extrinsic calibration and depth-based 3D reprojection, ensuring dense, cross-sensor labels.

In practice

• Combine DVS with 77GHz Radar for improved human detection.
• Utilize recurrent models for superior event-based object detection.
• Explore high-resolution DVS for enhanced spatial detection.

Topics

• NERVE Dataset
• Neuromorphic Vision
• Multi-Sensor Fusion
• Dynamic Vision Sensors
• mmWave Radar

Code references

• nerveproject-ut/nerve
• ultralytics/ultralytics

Best for: Research Scientist, Computer Vision Engineer, AI Scientist, Machine Learning Engineer, Robotics Engineer

Related on AIssential

• FS-DVS: A Frequency-Selective Dynamic Visual Sensing Paradigm for Enhancing Information Completeness — FS-DVS uses a learnable pre-trigger spatial filter to mimic RGCs, enhancing event camera data completeness and noise resilience.
• IEEE Transactions on Neural Networks and Learning Systems, Volume 37, Issue 5, May 2026 — The volume showcases diverse advancements in neural networks, from foundational surveys to specialized applications and novel architectures.
• Depth-Semantic Alignment and Affinity-Guided Fusion for Structured Radar Point Cloud Generation — Fusing vision and radar data improves sparse, noisy millimeter-wave radar point clouds for perception tasks.
• VGGT-$Ω$ — Feed-forward reconstruction models scale predictably with data and model size, offering geometry-aware features.
• Efficient Sensor Fusion for Gesture Recognition on Resource-Constrained Devices — Sensor fusion of low-resolution ToF and IR enables efficient, private gesture recognition on resource-constrained devices.
• SNNF: An SNN-based Near-Sensor Noise Filter for Dynamic Vision Sensors — SNNF efficiently filters DVS noise using a compact binary representation and a hardware-optimized Spiking Neural Network.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.