UAV-OVVIS: Unmanned Aerial Vehicles Also Need Open-Vocabulary Video Instance Segmentation

2026-07-09 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

UAV-OVVIS introduces a new task: Unmanned Aerial Vehicle Open-Vocabulary Video Instance Segmentation, addressing limitations in current UAV video perception. Existing methods struggle with flexible queries and fine-grained instance-level understanding in open scenarios, relying on box-level localization and predefined categories. UAV-OVVIS aims to discover targets in UAV videos using open-vocabulary queries, generating instance-level segmentation trajectories with globally consistent identities. To achieve this, the AeroTrack framework is proposed, a training-free unified system that reuses existing visual foundation models. AeroTrack operates through periodic open-vocabulary detection, short-segment mask propagation, and cross-segment identity unification. The researchers instantiated five AeroTrack variants and developed AeroVIS, an evaluation benchmark comprising 9 UAV object categories and 8,279 trajectories. Experiments confirm AeroTrack's substantial outperformance over general video instance segmentation methods in UAV contexts, demonstrating robust open-vocabulary generalization.

Key takeaway

For Computer Vision Engineers developing UAV perception systems, if you require flexible, fine-grained instance-level understanding in open scenarios, AeroTrack provides a robust, training-free solution. This framework allows you to discover targets using open-vocabulary queries and obtain consistent segmentation trajectories, overcoming limitations of predefined categories. Consider evaluating AeroTrack and the AeroVIS benchmark to enhance your UAV video analysis capabilities for applications like traffic monitoring or emergency response.

Key insights

UAV-OVVIS enables open-vocabulary, instance-level video segmentation for drones by reusing existing visual foundation models.

Principles

• Utilize existing visual foundation models.
• Prioritize training-free frameworks for new tasks.
• Unify identities across video segments.

Method

AeroTrack employs periodic open-vocabulary detection, short-segment mask propagation, and cross-segment identity unification to process UAV videos.

In practice

• Apply AeroTrack for flexible UAV video analysis.
• Utilize AeroVIS benchmark for UAV-OVVIS research.
• Integrate open-vocabulary segmentation into traffic monitoring.

Topics

• Unmanned Aerial Vehicles
• Video Instance Segmentation
• Open-Vocabulary Detection
• AeroTrack
• AeroVIS
• Computer Vision

Best for: Research Scientist, AI Scientist, Computer Vision Engineer, Robotics Engineer

Related on AIssential

• UAV-OVVIS: Unmanned Aerial Vehicles Also Need Open-Vocabulary Video Instance Segmentation — UAV-OVVIS enables flexible, fine-grained instance segmentation in UAV videos by reusing existing visual foundation models.
• Memory-Augmented Multimodal Large Language Models for Small Object Understanding in Streaming Aerial Videos — Language-guided aerial perception of tiny objects in streaming video requires specialized MLLMs and pixel-level datasets.
• VocaDet: Sample-Driven Open-Vocabulary Object Detection and Segmentation via Visual Tokenization and Vector Database Retrieval — VocaDet enables scalable open-vocabulary object detection by learning from samples via visual tokenization and vector database retrieval, eliminating retraining.
• RIM: A Retrieval-In-Matching Framework for Cross-Domain Global Visual Localization of UAVs — A Retrieval-In-Matching framework efficiently localizes UAVs in GNSS-challenged environments by reusing a single DINOv2-B backbone for both global retrieval and local description.
• Meta's SAM 3 (Free + Open Source) — SAM 3 enables text-prompted, open-source video object segmentation for easy analysis.
• HDST-GNN: Heterogeneous Dynamic Spatiotemporal Graph Neural Networks for Multi-Object Tracking in UAV Aerial Imagery — HDST-GNN improves multi-object tracking in UAV imagery by modeling heterogeneous object states and dynamic spatial context.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.