O3N: Omnidirectional Open-Vocabulary Occupancy Prediction

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Computer Vision · Depth: Expert, extended

Summary

O3N, a novel purely visual, end-to-end framework, introduces omnidirectional open-vocabulary occupancy prediction for autonomous agents and embodied intelligence. Developed by Hunan University and Zhejiang University, O3N addresses limitations of existing 3D occupancy methods by processing 360° visual inputs and predicting unseen object categories. It features the Polar-spiral Mamba (PsM) module for continuous spatial representation, the Occupancy Cost Aggregation (OCA) module for unifying geometric and semantic supervision, and Natural Modality Alignment (NMA) to harmonize "pixel-voxel-text" representations. O3N achieves state-of-the-art performance, with 16.54 mIoU and 21.16 Novel mIoU on QuadOcc, and 24.25 mIoU on Human360Occ, demonstrating strong cross-scene generalization and semantic scalability. The model runs at 9.41 FPS with 4.97 GB memory, trained for 25 epochs on 4 NVIDIA RTX 3090 GPUs with a total batch size of 4.

Key takeaway

For AI Engineers developing autonomous agents or embodied robotics, O3N offers a robust solution for 360° scene understanding with open-vocabulary capabilities. You should consider integrating its Polar-spiral Mamba, Occupancy Cost Aggregation, and Natural Modality Alignment modules to enhance your models' ability to perceive novel objects and generalize across diverse environments. This framework provides superior performance on real-world and simulated datasets, improving safety and adaptability in complex, dynamic settings.

Key insights

O3N unifies omnidirectional vision with open-vocabulary 3D occupancy prediction for comprehensive scene understanding.

Principles

• Polar-spiral voxel topology improves 360° spatial context modeling.
• Cost aggregation refines voxel-text similarity for open-vocabulary learning.
• Gradient-free alignment bridges modality gaps for robust generalization.

Method

O3N processes omnidirectional RGB images through a dual-branch Polar-spiral Mamba, aggregates occupancy costs with spatial and class reasoning, and aligns modalities via a gradient-free iterative process.

In practice

• Use PsM for continuous 360° spatial representation.
• Apply OCA to unify geometric and semantic supervision.
• Implement NMA for "pixel-voxel-text" consistency.

Topics

• Omnidirectional Perception
• Open-Vocabulary Occupancy
• 3D Occupancy Prediction
• Polar-spiral Mamba
• Modality Alignment
• Autonomous Agents

Code references

• MengfeiD/O3N

Best for: Computer Vision Engineer, Research Scientist, AI Scientist, AI Engineer, Robotics Engineer

Related on AIssential

• Occ-VLM: Occupancy Grounded Vision Language Model for Indoor Scene Understanding — Occ-VLM unifies 2D semantics and 3D geometry for scene understanding using only RGB images and occupancy as a spatial prior.
• H-OmniStereo: Zero-Shot Omnidirectional Stereo Matching with Heading-Aligned Normal Priors — H-OmniStereo improves omnidirectional stereo matching via a large synthetic dataset and a heading-aligned normal estimator.
• 🦪OccAny: Universal 3D Occupancy🦪 👉OccAny by Valeo is a novel unified framework for... — OccAny is a unified framework for generalized, unconstrained urban 3D occupancy prediction.
• Robust Onion: Peeling Open Vocab Object Detectors Under Noise — Open Vocabulary Object Detector robustness is primarily governed by vision backbones and image domain, with pretraining and annotations having minimal impact.
• Compact Object-Level Representations with Open-Vocabulary Understanding for Indoor Visual Relocalization — OpenReLoc uses object-level representations and open-vocabulary semantics for robust indoor visual relocalization.
• OmniLiDAR: A Unified Diffusion Framework for Multi-Domain 3D LiDAR Generation — OmniLiDAR unifies 3D LiDAR generation across diverse sensing domains using a text-conditioned diffusion framework.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.