facebookresearch / facebookresearch/sam2

Compute mask embeddings and similarities

Open
#405 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
19.9k
Forks
2.5k
PR merge metrics
No merged PRs in 30d

Description

HI,

I wonder what is the proper way to compute the embedding of a manually prompted mask (e.g with a few points), in order to do similarity matching with automatically labelled masks in the rest of the image and even other images.

In SAMv1 paper "D.6. Probing the Latent Space of SAM", it was said
" we compute mask embeddings by extracting an image embedding from SAM from an image crop around a mask and its horizontally flipped version, multiplying the image embedding by the binary mask, and averaging over spatial locations.".
This is also discussed in https://github.com/facebookresearch/segment-anything/issues/283

Should we keep the same protocol for SAM v2 ?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.