Memory-Augmented Multimodal Large Language Models for Small Object Understanding in Streaming Aerial Videos

2026-07-22 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

A new research effort introduces SkyAnchor, a Memory-Augmented Multimodal Large Language Model (MLLM) designed for small object understanding in streaming aerial videos from Unmanned Aerial Vehicles (UAVs). This model addresses two key challenges in online language-guided aerial perception: the tiny size of targets viewed from the air and the need for continuous past-frame context in resource-constrained streaming environments. SkyAnchor incorporates a Semantics-Aware Token Router to preserve fine-grained details of small targets despite visual compression, and a Hierarchical Memory Bank to maintain consistent target understanding across continuous video streams. Complementing this, the project also presents DroneEyes, the first pixel-level, open-vocabulary referring-segmentation dataset for tiny aerial targets, comprising 2,140 high-definition videos and 176,623 pairs with dense per-frame masks.

Key takeaway

For Computer Vision Engineers developing real-time object perception for UAVs, you should consider integrating memory-augmented MLLM architectures. Implement a semantics-aware token router to preserve tiny object details and a hierarchical memory bank to manage streaming context efficiently. This approach directly addresses the challenges of small target visibility and continuous online processing, improving the robustness and accuracy of your aerial perception systems. Explore the DroneEyes dataset for training and benchmarking your models.

Key insights

Memory-augmented MLLMs and specialized datasets improve small object understanding in streaming aerial video.

Principles

• Fine-grained detail preservation is crucial for tiny aerial targets.
• Online streaming perception requires efficient context retention.
• Dedicated datasets are vital for specialized MLLM training.

Method

SkyAnchor uses a Semantics-Aware Token Router for small-target detail preservation and a Hierarchical Memory Bank for consistent understanding in streaming MLLM processing.

In practice

• Develop MLLMs with token routing for tiny object focus.
• Implement hierarchical memory for streaming context.
• Utilize DroneEyes for aerial referring-segmentation training.

Topics

• Multimodal Large Language Models
• UAV Perception
• Small Object Detection
• Streaming Video Analysis
• DroneEyes Dataset
• Memory Augmentation

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Memory-Augmented Multimodal Large Language Models for Small Object Understanding in Streaming Aerial Videos — MLLMs for UAVs need specialized visual token routing and memory banks to handle tiny objects in streaming video.
• Memory-Augmented Multimodal Large Language Models for Small Object Understanding in Streaming Aerial Videos — Language-guided aerial perception of tiny objects in streaming video requires specialized MLLMs and pixel-level datasets.
• Watch, Remember, Reason: Human-View Video Understanding with MLLMs — The human-view perspective unifies MLLM video understanding through watching, remembering, and reasoning capabilities.
• Leveraging Large Vision Model for Multi-UAV Co-perception in Low-Altitude Wireless Networks — A novel framework optimizes multi-UAV perception by sparsifying visual data and using LVMs with DRL for efficient communication.
• Beyond Scalar Distances: Semantic Attribute Gradients from Frozen MLLMs for Visual Embeddings — SAGA uses frozen MLLMs to provide attribute-resolved supervision for vision encoders, moving beyond scalar distances.
• LOCUS: Local Visual Cue Search for Enhancing Fine-Grained Perception in Multimodal Large Language Models — LOCUS enhances MLLM fine-grained perception by internalizing local visual cue search via a training-time proxy task.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.