Instance-Enriched Semantic Maps for Visual Language Navigation

2026-07-14 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Computer Vision · Depth: Expert, quick

Summary

Instance-Enriched Semantic Maps (IESM) is a novel framework designed to enhance Visual Language Navigation (VLN) for embodied agents in complex indoor environments. Addressing limitations in existing systems regarding instance-level object detail and query robustness, IESM introduces three key contributions. First, it employs instance-level two-and-a-half-dimensional (2.5D) rich information mapping, constructing maps from color and depth observations using open-vocabulary panoptic segmentation. This method preserves vertical distinctions, captures small objects, and stores diverse semantic attributes and natural language captions with room-level context. Second, IESM features robust query processing via LLM-based target selection, dynamically routing queries to type-specialized experts and fusing their outputs for consistent goal selection across varied query formulations. Third, it offers a storage-efficient semantic representation, achieving approximately 96% reduction compared to three-dimensional (3D) scene-graph approaches while maintaining spatial information. The 2.5D representation surpasses the 3D baseline by over 27% in prediction-normalized Area Under the Curve (AUC), and the method improves object retrieval by over 17% and navigation success by over 23% compared to the baseline.

Key takeaway

For Robotics Engineers developing embodied agents for complex indoor navigation, your current systems may lack the instance-level detail and query robustness needed for reliable operation. You should consider integrating Instance-Enriched Semantic Maps (IESM) to achieve over 17% improvement in object retrieval and over 23% in navigation success. This approach, leveraging 2.5D mapping and LLM-based query processing, also offers a 96% reduction in storage compared to 3D scene-graphs, optimizing resource use for your agents.

Key insights

Instance-Enriched Semantic Maps (IESM) enhance Visual Language Navigation through detailed 2.5D mapping, robust LLM-driven query processing, and efficient spatial representation.

Principles

• Instance-level mapping significantly improves environmental understanding.
• LLM-based expert routing enhances query robustness and consistency.
• 2.5D representations offer a balance of detail and storage efficiency.

Method

Constructs 2.5D maps from color and depth using open-vocabulary panoptic segmentation, then processes queries via LLM-based target selection with dynamic expert routing and score-level fusion.

In practice

• Integrate 2.5D mapping for fine-grained object detail.
• Employ LLM-driven expert systems for diverse query handling.
• Utilize 2.5D representations to reduce map storage by 96%.

Topics

• Visual Language Navigation
• Embodied AI
• Semantic Mapping
• Panoptic Segmentation
• Large Language Models
• 2.5D Representation

Best for: Research Scientist, AI Scientist, Robotics Engineer, Computer Vision Engineer

Related on AIssential

• From Language to Navigation Goals: A Vision-Language Approach for Semantic Navigation of Mobile Robots Using RGB-D Perception — Robots can navigate intuitively by interpreting natural language commands and using RGB-D perception.
• Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator — Neural simulation from real-world captures can scale embodied navigation data and improve real-world transfer.
• GEM-Occ: From Visual Geometry Evidence to Embodied Semantic Occupancy Memory — Persistent semantic occupancy mapping for embodied agents requires hierarchical memory and causal fusion of transient visual evidence.
• FSD-VLN: Fast-Slow Dual-System Modeling for Aerial Long-Horizon Vision-Language Navigation — Decoupling semantic reasoning and flight control via a fast-slow dual-system significantly enhances aerial Vision-Language Navigation performance.
• ABot-N1: Toward a General Visual Language Navigation Foundation Model — Decoupling cognition from control via a slow-fast architecture improves VLN model generality and interpretability.
• VTM-Nav: Hierarchical Visual-Topological Memory for Cross-Episode Object-Goal Navigation — VTM-Nav uses hierarchical visual-topological memory for robust, training-free cross-episode object-goal navigation.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.