facebookresearch / facebookresearch/segment-anything

Discrepancy results between online demo and SamAutomaticMaskGenerator

Open
#716 7 comments 5 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
54.9k
Forks
6.4k
PR merge metrics
No merged PRs in 30d

Description

Hi,

I have been trying out SAM in the online demo and it works great using the "Everything" button:

**Image:**
![image](https://github.com/facebookresearch/segment-anything/assets/4972463/81a57237-08fb-4675-82b5-3e69a0b28eef)

**Result demo:**
![results_demo](https://github.com/facebookresearch/segment-anything/assets/4972463/34f4b35f-55fd-400c-8648-bd87587cfd98)

I then tried to recreate these results with the code using the SamAutomaticMaskGenerator and a vit-h (default) model. See code:

```
# open the image
image = cv2.imread('image.png')
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

# initialize SAM vit-h and a mask_generator
sam_vit_h = sam_model_registry['vit_h'](checkpoint='checkpoints/sam_vit_h_4b8939.pth').to(device='cuda')
mask_generator_vit_h = SamAutomaticMaskGenerator(sam_vit_h)

# generate masks
masks = mask_generator_vit_h.generate(image)

# plot
plt.figure(figsize=(10, 10))
plt.imshow(image)
show_anns(masks, one_color=False)
plt.axis('off')
plt.savefig('result_code.png', bbox_inches='tight', pad_inches=0)
plt.show()
```

However,` I get a different result. Large portion of the background is segmented and there is also many more small regions detected.

**Result code:**
![results_code](https://github.com/facebookresearch/segment-anything/assets/4972463/29f32f2f-0b93-4622-ba2b-187bce016daf)

For my application the demo result is preferred. If anyone knows how to recreate the demo results, it would be greatly appreciated if its shared!

Contributor guide

Open the contributing guide

Research direction

Start with the provided SamAutomaticMaskGenerator.generate(image) snippet and compare its defaults with the online demo's "Everything" action. Re-run the example using image.png and inspect the demo and code mask outputs; done means identifying the setting or processing difference that explains the discrepancy, or documenting why the results cannot match.

Written by the indexing model from the issue text.

Assessment

Tech stack
matplotlib, opencv, python, pytorch
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.