Prompt-Calibrated SAM 3 for Open-Vocabulary Remote Sensing Semantic Segmentation

2026-07-23 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Environmental Science & Earth Systems · Depth: Expert, long

Summary

ProC-SAM3, a training-free framework, improves SAM 3's prompt interface for open-vocabulary semantic segmentation (OVSS) in remote sensing images. It addresses key issues like insufficient semantic coverage from single class-name prompts, redundant online text encoding from multiple prompts, and noisy activations from direct aggregation. The framework achieves an average mIoU of 56.1% on eight benchmarks, outperforming the previous best training-free method by 3.9 percentage points. ProC-SAM3 integrates three components: Offline Prompt-pool Construction (OPC) for GT-free prompt calibration using MLLM-generated candidates and expansion constraints; Prompt-Optimized Batched Inference (POBI) for caching and reusing text embeddings; and Presence-Guided Residual Fusion (PGRF) for robust mask aggregation, preserving fine-grained activations.

Key takeaway

For Machine Learning Engineers adapting SAM 3 for open-vocabulary semantic segmentation in remote sensing, you should integrate ProC-SAM3's training-free approach. Its offline prompt pool construction, cached text embeddings, and presence-guided residual fusion address common issues like semantic drift and noisy activations. This method improves average mIoU by 3.9 percentage points, offering a robust path to higher accuracy and inference efficiency for your geospatial scene interpretation tasks.

Key insights

Calibrating SAM 3's prompt interface with offline construction, cached embeddings, and guided fusion significantly improves remote sensing OVSS.

Principles

• Single class-name prompts lack semantic coverage for complex categories.
• Unconstrained MLLM prompt expansion introduces semantic drift.
• Gating unreliable decoder outputs improves prediction quality.

Method

ProC-SAM3 constructs an offline prompt pool via MLLM generation and constraint-based refinement, caches text embeddings for batched decoding, and fuses mask responses using presence-guided residual gating and pixel-wise max-pooling.

In practice

• Use MLLMs for candidate prompt generation, then refine with constraints.
• Cache text embeddings to avoid redundant online encoding.
• Apply pixel-wise max-pooling for small object preservation.

Topics

• Open-Vocabulary Semantic Segmentation
• Remote Sensing
• SAM 3
• Prompt Engineering
• Multimodal Large Language Models
• Inference Optimization
• Mask Aggregation

Code references

• YanghuiSong/ProC-SAM3

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Promptable Concept Segmentation from Above: Evaluating SAM 3's Zero-Shot and One-Shot Capabilities in Remote Sensing — SAM 3 shows promise in remote sensing but suffers from ground-level semantic bias and sub-pixel resolution limits.
• SAM 3: Concept-Based Visual Understanding and Segmentation — SAM 3 unifies detection, segmentation, and tracking of open-vocabulary concepts using text or visual prompts.
• ReA-OVCD: Reliability-Aware Open-Vocabulary Change Detection via Semantic and Spatial Refinement — ReA-OVCD refines open-vocabulary change detection by collaboratively addressing semantic ambiguity and spatial inconsistency through a training-free, two-module approach.
• Farmland Extent and Visible Boundary Mapping from 1 m NAIP Imagery Using Residual U-Net and Text-Prompted SAM 3 Refinement — Combining a specialized U-Net with text-prompted SAM 3 effectively maps farmland boundaries from NAIP imagery, improving accuracy on complex terrains.
• ProCal: Inference-Time Proposal Calibration for Open-Vocabulary Object Detection — ProCal improves open-vocabulary object detection by calibrating VLM classification scores with foreground/background awareness at inference time.
• JL1-CC&QA: Extending the JL1-CD Benchmark with Change Captioning and Question Answering — Remote sensing change detection benefits from semantic layers like captioning and QA to understand "what" and "why" changes occur.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.