SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Scene Segmentation

Mutian Xu1, Xingyilang Yin1, Lingteng Qiu1, Yang Liu3, Xin Tong3, Xiaoguang Han1, 2†
1SSE, CUHKSZ    2FNii, CUHKSZ    3Microsoft Research Asia
†Corresponding Author
arXiv Code
MY ALT TEXT

We introduce SAMPro3D for zero-shot 3D indoor scene segmentation.

Abstract

We introduce SAMPro3D for zero-shot 3D indoor scene segmentation. Given the 3D point cloud and multiple posed 2D frames of 3D scenes, our approach segments 3D scenes by applying the pretrained Segment Anything Model (SAM) to 2D frames. Our key idea involves locating 3D points in scenes as natural 3D prompts to align their projected pixel prompts across frames, ensuring frame-consistency in both pixel prompts and their SAM-predicted masks. Moreover, we suggest filtering out low-quality 3D prompts based on feedback from all 2D frames, for enhancing segmentation quality. We also propose to consolidate different 3D prompts if they are segmenting the same object, bringing a more comprehensive segmentation. Notably, our method does not require any additional training on domain-specific data, enabling us to preserve the zero-shot power of SAM. Extensive qualitative and quantitative results show that our method consistently achieves higher quality and more diverse segmentation than previous zero-shot or fully supervised approaches, and in many cases even surpasses human-level annotations.


Key Idea

MY ALT TEXT

The comparison of our key idea and others, as well as how they impact segmentation results.


Method Overview

MY ALT TEXT

An overview of our framework, with a primary focus on "prompt". We locate initial 3D prompts in input scenes, then project them onto 2D frames, where SAM is performed to obtain 2D segmentation masks. Later, the initial 3D prompts and their corresponding 2D masks are filtered and consolidated. Finally, we project all input points onto each segmented frame to obtain the 3D segmentation result.

Animated Qualitative Comparison

Qualitative Comparison

BibTeX

@article{xu2023sampro3d,
        title={SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Scene Segmentation}, 
        author={Mutian Xu and Xingyilang Yin and Lingteng Qiu and Yang Liu and Xin Tong and Xiaoguang Han},
        year={2023},
        journal = {arXiv preprint arXiv:2311.17707}
  }