Grounded SAM 2: From Open-Set Detection to Segmentation and Tracking

2026-01-19 · Source: PyImageSearch · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Advanced, extended

Summary

This tutorial details Grounded SAM 2, a vision-language pipeline that extends Grounding DINO's open-set object detection with pixel-level segmentation and video tracking. While Grounding DINO identifies objects with bounding boxes using natural language prompts, it lacks spatial precision. Grounded SAM 2 addresses this by integrating SAM 2, which generates precise segmentation masks and maintains temporal consistency across video frames via a streaming-memory transformer. The pipeline first uses Grounding DINO for language-driven detection, then passes the detected bounding boxes to SAM 2 for high-precision segmentation, and finally tracks these segmented objects across video frames. The article provides a step-by-step implementation guide, including environment setup, model checkpoint downloads, and a `run_tracking` function, culminating in a Gradio interface for interactive video processing and visualization.

Key takeaway

For Machine Learning Engineers building advanced video analytics or robotics applications, Grounded SAM 2 offers a robust solution for precise object understanding. You should integrate this pipeline to move beyond coarse bounding box detection, enabling pixel-accurate segmentation and consistent object tracking across video frames using natural language prompts. This approach simplifies data annotation and enhances capabilities in domains like medical imaging or autonomous driving.

Key insights

Grounded SAM 2 combines language-driven detection with pixel-level segmentation and video tracking for comprehensive visual understanding.

Principles

• Segmentation offers higher spatial precision than bounding boxes.
• Promptable segmentation models generalize to unseen categories.
• Streaming-memory transformers ensure temporal consistency in video segmentation.

Method

The Grounded SAM 2 pipeline cascades a grounding model (e.g., Grounding DINO) for detection, then SAM 2 for promptable segmentation, and finally layers on tracking and heuristics for video processing.

In practice

• Use `pip install sam2` for SAM 2 integration.
• Employ `supervision` for efficient annotation utilities.
• Utilize `hf_hub_download` for model weight retrieval.

Topics

• Grounded SAM 2
• Open-Set Segmentation
• Video Object Tracking
• Vision-Language Models
• Grounding DINO

Code references

• IDEA-Research/Grounded-SAM-2
• IDEA-Research/Grounded-SAM-2

Best for: Machine Learning Engineer, Deep Learning Engineer, AI Student

Related on AIssential

• SSP-SAM: SAM with Semantic-Spatial Prompt for Referring Expression Segmentation — SSP-SAM enhances SAM's language understanding for segmentation by integrating semantic and spatial cues via a novel prompt encoder.
• Foresee-to-Ground: From Predictive Temporal Perception to Evidence-Driven Reasoning for Video Temporal Grounding — F2G improves video temporal grounding by decoupling event identification from precise boundary measurement using citable evidence.
• Grounding Qwen3-VL Detection with SAM2 — Combining Qwen3-VL's detection with SAM2's segmentation enables robust, natural language-driven object grounding.
• Grounded 3D-Aware Spatial Vision-Language Modeling — GR3D integrates explicit 2D, implicit 2D, and monocular 3D grounding to enhance spatial understanding in vision-language models.
• SAM 3 for Video: Concept-Aware Segmentation and Object Tracking — SAM3 unifies detection, segmentation, and tracking in video by maintaining streaming memory and tracking state across frames.
• Advanced SAM 3: Multi-Modal Prompting and Interactive Segmentation — SAM 3 offers flexible multi-modal and interactive prompting for precise, context-aware image segmentation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by PyImageSearch.