UAV-OVVIS: Unmanned Aerial Vehicles Also Need Open-Vocabulary Video Instance Segmentation

2026-07-23 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

A new task, UAV Open-Vocabulary Video Instance Segmentation (UAV-OVVIS), is introduced to enable flexible, fine-grained instance-level dynamic understanding in Unmanned Aerial Vehicle (UAV) videos. This task discovers targets based on open-vocabulary queries and outputs instance-level segmentation trajectories with globally consistent identities. To address the scarcity of instance-level annotations in UAV scenarios, the authors propose AeroTrack, a training-free unified framework. AeroTrack reuses existing visual foundation models (VFMs) by decoupling UAV-OVVIS into periodic open-vocabulary detection, short-segment mask propagation, and cross-segment identity unification via Lifecycle-aware ID Association (LIA). The framework instantiates five variants and is evaluated on AeroVIS, a new benchmark containing 9 UAV object categories and 8,279 trajectories. Experiments show AeroTrack substantially outperforms existing general video instance segmentation methods in UAV scenarios, demonstrating strong open-vocabulary robustness and generalization, while also offering significant memory efficiency for long videos.

Key takeaway

For Machine Learning Engineers developing UAV video analytics, existing general video instance segmentation methods often fall short in complex aerial scenarios. You should consider adopting the AeroTrack framework to achieve robust open-vocabulary instance segmentation. Its training-free approach, leveraging existing visual foundation models, significantly improves performance and memory efficiency for long, dense UAV videos. Implement periodic detection and Lifecycle-aware ID Association to maintain consistent object identities across segments.

Key insights

UAV-OVVIS enables flexible, fine-grained instance segmentation in UAV videos by reusing existing visual foundation models.

Principles

• Decouple complex tasks into cooperative VFM components.
• Periodic detection and short-segment propagation manage resources.
• Lifecycle-aware ID association unifies fragmented identities.

Method

AeroTrack periodically detects objects on key frames, propagates instance masks within short segments, and unifies cross-segment identities using Lifecycle-aware ID Association (LIA).

In practice

• Use YOLO-World or Grounding DINO as Recognizers.
• Employ SAM2 or SAM3 as Segmenters for mask propagation.
• Set detection refresh interval to Δ=25 frames for efficiency.

Topics

• UAV Video Analytics
• Open-Vocabulary Segmentation
• Visual Foundation Models
• AeroTrack Framework
• Instance Segmentation
• Lifecycle-aware ID Association

Code references

• Dmygithub/AeroTrack

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• UAV-OVVIS: Unmanned Aerial Vehicles Also Need Open-Vocabulary Video Instance Segmentation — UAV-OVVIS enables open-vocabulary, instance-level video segmentation for drones by reusing existing visual foundation models.
• Mind the Gap: Disentangling Performance Bottlenecks in Video Instance Segmentation — Tracking instability, not classification or representation, is the primary bottleneck for online Video Instance Segmentation methods.
• VocaDet: Sample-Driven Open-Vocabulary Object Detection and Segmentation via Visual Tokenization and Vector Database Retrieval — VocaDet enables open-vocabulary object recognition by converting visual features into discrete tokens for efficient vector database retrieval.
• Open-World Video Segmentation — Savvy and OGA advance open-world video segmentation by combining novel tracking with granularity-aware evaluation.
• Memory-Augmented Multimodal Large Language Models for Small Object Understanding in Streaming Aerial Videos — MLLMs for UAVs need specialized visual token routing and memory banks to handle tiny objects in streaming video.
• SegmentAnyTreeV2: Scaling Transformer-Based Tree Instance Segmentation Across Sensors, Platforms, and Forests — SegmentAnyTreeV2 offers robust, scalable tree instance segmentation for forest point clouds using a Transformer-based architecture.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.