TinySAM 2: Extreme Memory Compression for Efficient Track Anything Model

2026-05-18 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, medium

Summary

TinySAM 2 is a lightweight video segmentation model designed to enhance the deployment efficiency of Segment Anything Model 2 (SAM 2) in practical applications. SAM 2, a foundational model for video segmentation, utilizes a memory bank mechanism for tasks like semi-supervised video object segmentation and tracking. However, its complex multi-stage image encoder and memory module create deployment bottlenecks. TinySAM 2 addresses this by introducing a memory quality management mechanism to select high-informative historical frames and a joint-spatial-temporal token compression method. This compression uses average pooling for spatial redundancy and token-level similarity for temporal selection. Additionally, TinySAM 2 incorporates RepViT as a lightweight image encoder to reduce parameters. Experiments on DAVIS and SA-V datasets show TinySAM 2 achieves 90% of SAM 2.1's performance with only 7% memory tokens and 3% training data, significantly reducing parameter count, computational load, and deployment costs.

Key takeaway

For AI Engineers and Research Scientists deploying video segmentation models, TinySAM 2 offers a compelling solution to overcome the memory and computational bottlenecks of SAM 2. By adopting its memory quality management and joint-spatial-temporal token compression techniques, you can achieve near state-of-the-art performance with significantly reduced resource requirements, enabling broader application on resource-constrained devices. Consider evaluating TinySAM 2 for your next video segmentation project to optimize deployment efficiency.

Key insights

TinySAM 2 significantly compresses SAM 2 for efficient video segmentation while retaining high performance.

Principles

• Prioritize informative historical frames for memory efficiency.
• Compress tokens in both spatial and temporal domains.
• Utilize lightweight encoders to reduce model parameters.

Method

TinySAM 2 employs memory quality management to select informative frames, joint-spatial-temporal token compression via average pooling and similarity measurement, and integrates RepViT as a lightweight image encoder.

In practice

• Implement average pooling for spatial token compression.
• Use token-level similarity for temporal memory selection.
• Integrate RepViT for reduced model parameters.

Topics

• TinySAM 2
• SAM 2
• Video Segmentation
• Memory Compression
• Token Compression

Code references

• xinghaow99/BitStack
• xiaohuiwang000/UltraDelta
• LeanModels/DFloat11

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• X2SAM: Any Segmentation in Images and Videos — X2SAM unifies image and video segmentation via an MLLM and Mask Memory for consistent, prompt-driven mask generation.
• We’re releasing SAM 3.1: a drop-in update to SAM 3 that introduces object multiplexing to significantly improve video processing efficiency without sacrificing accuracy. We’re sharing this update with the community to help make high-performance applications f - x.com — SAM 3.1 improves video processing efficiency via object multiplexing, enabling high-performance AI on smaller hardware.
• ProtoKV: Streaming Video Understanding under Delayed Query with Summary-State Memory — ProtoKV uses a summary-state memory with a prototype bank to efficiently handle delayed queries in streaming video understanding.
• Exploiting Semantic and Pixel Representations for Ultra-Low Bitrate Image Compression — Ultra-low bitrate image compression can achieve pixel-level fidelity by combining semantic and pixel representations in a diffusion model.
• SA-VIS: Sparse frame Annotations for training Video Instance Segmentation — SA-VIS enables high-performance video instance segmentation with significantly fewer annotations by propagating past-frame features.
• ActiveSAM: Image-Conditional Class Pruning for Fast and Accurate Open-Vocabulary Segmentation — ActiveSAM enables faster, more accurate open-vocabulary segmentation by dynamically pruning classes per image for SAM 3.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.