Memory-Augmented Multimodal Large Language Models for Small Object Understanding in Streaming Aerial Videos

2026-07-23 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Data Science & Analytics · Depth: Expert, extended

Summary

A new multimodal large language model (MLLM), SkyAnchor, and its accompanying dataset, DroneEyes, address challenges in language-guided aerial perception for tiny objects in streaming UAV video. DroneEyes is the first pixel-level, open-vocabulary referring-segmentation dataset for tiny aerial targets, featuring 2,140 high-definition videos and 176,623 annotated pairs. SkyAnchor incorporates a Semantics-Aware Token Router to preserve fine-grained details of small targets under a reduced visual-token budget and a Hierarchical Memory Bank for consistent object understanding across video streams. Experiments show SkyAnchor outperforms existing MLLMs on DroneEyes by 2x in GPT score for object description and 15.3% in region-and-contour accuracy for referring expression. It also achieves strong cross-domain generalization on the SkyFind benchmark, improving IoU@0.5 by 25.9% and IoU@mean by 95.5% on an unseen maritime test split. Optimized for deployment, SkyAnchor runs on an NVIDIA Jetson AGX Orin 64GB with a 3.05x speedup over vanilla PyTorch, enabling real-time UAV surveillance.

Key takeaway

For machine learning engineers developing UAV-based perception systems, you should prioritize models designed for streaming video and tiny object recognition. General MLLMs are insufficient; instead, consider architectures like SkyAnchor that use semantics-aware token routing and hierarchical memory to maintain target identity and detail. This approach enables robust, real-time, cloud-free deployment on edge hardware like the NVIDIA Jetson AGX Orin, crucial for effective aerial surveillance and operator assistance.

Key insights

Language-guided aerial perception of tiny objects in streaming video requires specialized MLLMs and pixel-level datasets.

Principles

• Tiny aerial objects demand pixel-accurate perception.
• Streaming video requires efficient past-frame context.
• General MLLMs often discard fine-grained details.

Method

SkyAnchor uses a Semantics-Aware Token Router for small-target preservation under token budget and a Hierarchical Memory Bank for consistent understanding in streams, combining object-level semantic memory and short-term tracking memory.

In practice

• Construct datasets with pixel-level, open-vocabulary annotations.
• Implement semantics-aware token routing for small objects.
• Decouple temporal memory into semantic and tracking layers.

Topics

• Multimodal LLMs
• UAV Perception
• Streaming Video
• Small Object Detection
• Referring Segmentation
• Edge AI

Best for: Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, Robotics Engineer

Related on AIssential

• Memory-Augmented Multimodal Large Language Models for Small Object Understanding in Streaming Aerial Videos — MLLMs for UAVs need specialized visual token routing and memory banks to handle tiny objects in streaming video.
• Memory-Augmented Multimodal Large Language Models for Small Object Understanding in Streaming Aerial Videos — Memory-augmented MLLMs and specialized datasets improve small object understanding in streaming aerial video.
• Leveraging Large Vision Model for Multi-UAV Co-perception in Low-Altitude Wireless Networks — A novel framework optimizes multi-UAV perception by sparsifying visual data and using LVMs with DRL for efficient communication.
• PixDLM: A Dual-Path Multimodal Language Model for UAV Reasoning Segmentation — UAV reasoning segmentation requires specialized dual-path models and high-resolution, reasoning-rich benchmarks to handle unique aerial imagery challenges.
• Learning Geometric Representations from Videos for Spatial Intelligent Multimodal Large Language Models — GeoVR fundamentally restructures MLLM's internal representations for 3D spatial intelligence using 2D video and 3D foundation model distillation, without inference overhead.
• Watch, Remember, Reason: Human-View Video Understanding with MLLMs — The human-view perspective unifies MLLM video understanding through watching, remembering, and reasoning capabilities.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.