Lean-SAM2: Target-Anchored Memory and Encoder Acceleration for SAM2

2026-07-23 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

Lean-SAM2 is a lightweight framework designed to enhance the Segment Anything Model 2 (SAM2) by addressing its heavy memory cross-attention overhead and redundant full-frame visual feature extraction, which hinder deployment in real-time scenarios. Existing efficiency methods often suffer performance degradation in complex segmentation with occlusions and distractors. Lean-SAM2 integrates three mechanisms: Target-Anchored Memory Pruning (TAMP) to safeguard target tokens, Temporal Condensation with Insurance Memory (TCIM) to condense historical context, and Target-Anchored Risk-Aware Routing (TARR) to selectively activate the image encoder. Evaluations on the LVOSv2 validation dataset show Lean-SAM2 achieves inference speedups of 1.412x and 1.417x on SAM2.1-Large and SAM2.1-Base+, respectively, while boosting J&F scores by 5.0% and 3.6% over Efficient-SAM2.

Key takeaway

For Machine Learning Engineers deploying SAM2 in real-time video object segmentation, Lean-SAM2 offers a robust solution to overcome efficiency bottlenecks and performance degradation in complex scenarios. You should consider integrating its target-anchored memory pruning and risk-aware routing mechanisms to achieve significant inference speedups, such as 1.417x on SAM2.1-Base+, while maintaining or boosting segmentation accuracy, particularly when dealing with occlusions and distractors. This approach ensures more reliable and efficient model deployment.

Key insights

Lean-SAM2 improves SAM2 efficiency and robustness in complex video segmentation by anchoring memory and routing to target identity.

Principles

• Anchor memory pruning to initial target identity.
• Condense historical context with visibility gating.
• Route encoder processing based on target similarity.

Method

Lean-SAM2 uses TAMP for intra-frame memory pruning, TCIM for inter-frame condensation with an insurance bank, and TARR for risk-aware, target-anchored image encoder routing.

In practice

• Use prompt-derived foreground anchors for robust token pruning.
• Archive high-confidence memory entries in an insurance bank.
• Trigger full-frame encoder refreshes during volatile tracking.

Topics

• Segment Anything Model 2 (SAM2)
• Video Object Segmentation
• Memory Pruning
• Encoder Acceleration
• Temporal Condensation
• Sparse Routing
• Real-time Inference

Code references

• DeawhaleQwQ/Lean-SAM2

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Lean-SAM2: Target-Anchored Memory and Encoder Acceleration for SAM2 — Lean-SAM2 improves SAM2 efficiency and accuracy by intelligently pruning memory, condensing temporal context, and selectively routing encoder activation.
• SAM 3 for Video: Concept-Aware Segmentation and Object Tracking — SAM3 unifies detection, segmentation, and tracking in video by maintaining streaming memory and tracking state across frames.
• TinySAM 2: Extreme Memory Compression for Efficient Track Anything Model — TinySAM 2 significantly compresses SAM 2 for efficient video segmentation while retaining high performance.
• SAM-MT: Real-Time Interactive Multi-Target Video Segmentation — SAM-MT enables real-time multi-target video segmentation by decoupling latency from target count using explicit queries and shared context.
• Efficient Tracking and Understanding Object Transformations — SAM2's multi-mask disagreement provides a lightweight, reactive signal for efficient object transformation detection.
• We’re releasing SAM 3.1: a drop-in update to SAM 3 that introduces object multiplexing to significantly improve video processing efficiency without sacrificing accuracy. We’re sharing this update with the community to help make high-performance applications f - x.com — SAM 3.1 improves video processing efficiency via object multiplexing, enabling high-performance AI on smaller hardware.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.