SAM-MT: Real-Time Interactive Multi-Target Video Segmentation

2026-07-09 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

SAM-MT is a novel model designed for real-time interactive multi-target video segmentation, building upon Segment Anything 2 (SAM2). Traditional video object segmentation (VOS) methods struggle with multi-target scenarios, experiencing reduced frame rates and increased latency as the number of objects grows. SAM-MT addresses this by transforming SAM2 into an interactive framework that decouples latency from the target count. It achieves real-time performance, delivering over 36 FPS for 10 targets, comparable to single-target baselines, while preserving SAM2's robust segmentation capabilities. The model employs explicit queries for individual targets, a shared global context, decoupled masked attention to prevent cross-target interference, and sparse memory for stable temporal evolution, alongside specialized strategies for occlusion handling and overlap prevention.

Key takeaway

For Computer Vision Engineers developing real-time interactive video applications, SAM-MT offers a solution to the performance bottlenecks of multi-target segmentation. You can now implement robust multi-object tracking and segmentation without the typical latency increase associated with more targets. Consider integrating SAM-MT, built on SAM2, to achieve over 36 FPS for 10 targets, ensuring your systems remain responsive and scalable.

Key insights

SAM-MT enables real-time multi-target video segmentation by decoupling latency from target count using explicit queries and shared context.

Principles

• Decouple latency from target count.
• Use explicit queries for distinct identities.
• Employ shared context for global understanding.

Method

SAM-MT uses explicit queries for individual targets, a shared global context, decoupled masked attention for identity distinction, and sparse memory for temporal stability, plus occlusion/overlap strategies.

In practice

• Segment multiple objects simultaneously.
• Maintain real-time performance with many targets.
• Handle occlusions in dynamic scenes.

Topics

• Video Object Segmentation
• Multi-Target Segmentation
• Real-Time Systems
• Segment Anything 2
• Computer Vision
• Latency Decoupling

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• 💋SAM-MT: Real-Time Multi-Target VOS💋 👉Fudan & Shangai unveil SAM-MT, an efficient... — SAM-MT offers efficient, robust interactive multi-target video segmentation without significant performance degradation as target count rises.
• SAM 3.1: Faster and More Accessible Real-Time Video Detection and Tracking With Multiplexing and Global Reasoning - AI at Meta — SAM 3.1 enhances real-time video object tracking via multiplexing, doubling processing speed and reducing GPU demands.
• Lean-SAM2: Target-Anchored Memory and Encoder Acceleration for SAM2 — Lean-SAM2 improves SAM2 efficiency and robustness in complex video segmentation by anchoring memory and routing to target identity.
• An Open-Source Benchmark and Baseline for Multi-temporal Referring Segmentation — Multi-temporal Referring Segmentation (MTRS) is a new task and benchmark for language-guided temporal change detection in images.
• Amodal SAM: A Unified Amodal Segmentation Framework with Generalization — Amodal SAM extends SAM's generalization to predict complete object shapes, including occluded regions, using synthetic data and novel learning.
• SAM 3 for Video: Concept-Aware Segmentation and Object Tracking — SAM3 unifies detection, segmentation, and tracking in video by maintaining streaming memory and tracking state across frames.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.