facebookresearch / facebookresearch/sam2
Compute mask embeddings and similarities
- Dominant language
- Jupyter Notebook
- Stars
- 19.9k
- Forks
- 2.5k
- PR merge metrics
- No merged PRs in 30d
Description
HI,
I wonder what is the proper way to compute the embedding of a manually prompted mask (e.g with a few points), in order to do similarity matching with automatically labelled masks in the rest of the image and even other images.
In SAMv1 paper "D.6. Probing the Latent Space of SAM", it was said
" we compute mask embeddings by extracting an image embedding from SAM from an image crop around a mask and its horizontally flipped version, multiplying the image embedding by the binary mask, and averaging over spatial locations.".
This is also discussed in https://github.com/facebookresearch/segment-anything/issues/283
Should we keep the same protocol for SAM v2 ?
Contributor guide
Assessment
This issue has not been assessed yet.