SAM 3D - AI at Meta

2025-07-21 · Source: ai.meta.com via Google News · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Computer Vision · Depth: Intermediate, quick

Summary

Meta has introduced SAM 3D, a suite of two models, SAM 3D Objects and SAM 3D Body, designed for accurate 3D reconstruction from single 2D images. SAM 3D Objects reconstructs detailed 3D geometry and texture of masked objects, robust to occlusion, and suitable for manipulation. SAM 3D Body estimates human body shape and pose, even with partial visibility, also enabling manipulation. Both models can jointly reconstruct multiple objects and people within a shared scene context. The architecture for SAM 3D Body uses a transformer-based encoder-decoder for pose and mesh parameters, while SAM 3D Objects employs two stages of DiTs for shape, pose, texture, and detail refinement. Meta claims SAM 3D achieves state-of-the-art performance across various benchmarks and is designed for practical applications like AR shopping on Facebook Marketplace, physical therapy, and robotics.

Key takeaway

For Machine Learning Engineers developing AR/VR applications or robotics, SAM 3D offers robust 3D reconstruction capabilities from single images. You should explore integrating SAM 3D Objects and SAM 3D Body to create interactive 3D experiences, such as visualizing products in real-world environments or enabling more sophisticated human-robot interaction. Consider leveraging the joint reconstruction feature for complex scene understanding.

Key insights

Meta's SAM 3D models reconstruct detailed 3D objects and human bodies from single 2D images.

Principles

• Single 2D images can yield high-fidelity 3D reconstructions.
• Joint reconstruction enhances scene understanding.

Method

SAM 3D Body uses a transformer encoder-decoder for human pose/mesh; SAM 3D Objects employs two DiT stages for object shape, pose, and texture refinement.

In practice

• Integrate 3D AR overlays for e-commerce visualization.
• Develop advanced robotics perception systems.

Topics

• SAM 3D
• 3D Reconstruction
• Human Mesh Recovery
• Object Reconstruction
• Transformer Architecture

Code references

• facebookresearch/sam-3d-objects
• facebookresearch/sam-3d-body

Best for: Machine Learning Engineer, Research Scientist, AI Scientist, Computer Vision Engineer, AI Engineer

Related on AIssential

• Make 3D models from ANYTHING (Free and Open Source) — Meta's SAM 3D is an open-source, open-weights model that converts 2D images into 3D models, offering significant utility across various applications.
• SAM3D-Phys: Towards Multi-Object Interactive Simulation in Real World — SAM3D-Phys recovers complete, simulatable 3D object geometry from partial real-world scene observations for physics-based interaction.
• SAM 3 - AI at Meta — SAM 3 offers state-of-the-art object segmentation and tracking across images and videos using diverse text and visual prompts.
• 🍧10,000× faster SAM-3D🍧 👉Fast SAM 3D Body achieves up to 10.9× speedup, over 10,000×... — A novel method, "Fast SAM 3D Body," significantly accelerates 3D body reconstruction, achieving an impressive 10.9× speedup and over 10,000× faster MHR-to-SMPL conversion.
• Learned Radius Estimation for UDF-Based Point Cloud Reconstruction — Learning to adapt the support radius dynamically improves UDF-based point cloud reconstruction accuracy, especially for complex geometries.
• CHOIR: Contact-aware 4D Hand-Object Interaction Reconstruction — CHOIR reconstructs 4D hand-object interactions from monocular video, explicitly using contact as a coupling signal for improved consistency.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by ai.meta.com via Google News.