facebookresearch / facebookresearch/segment-anything

Performance Discrepancy Between SAM Model Demo and GitHub Code

Open
#762 12 comments 7 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
54.9k
Forks
6.4k
PR merge metrics
No merged PRs in 30d

Description

Hi, I am working on segmenting car bodies in images using the Meta SAM model. I am facing a significant difference in performance between the UI demo on the official website and the code provided on the GitHub repository. UI demo performed remarkably well with just 1-2 clicks, however, when I attempted to use the code, results are very different and bad. Despite of providing multiple points, the results were not up to the mark as compared to the demo.

Using SAM Model Version:- **"vit_h"**
Used predictor_example file:- **notebooks/predictor_example.ipynb**

**Examples:**
**Image 1:**
Original Image:
![image3](https://github.com/facebookresearch/segment-anything/assets/92079088/7c54a8a7-0f63-4277-99ef-f09eecec3680)

UI Demo Segmentation: - Performed well with 4 foreground points and 3 background points.
![resized_sam_ui_3](https://github.com/facebookresearch/segment-anything/assets/92079088/a8bb7a03-5fb0-45dd-af8a-751981efe147)

My Code Segmentation: - Poor results with the same point placement.
![code_output_3](https://github.com/facebookresearch/segment-anything/assets/92079088/8577e8d3-1e2a-4425-868b-d0df99016387)

**Image 2:**
Original Image:
![image2](https://github.com/facebookresearch/segment-anything/assets/92079088/bb3d6fc7-364f-458a-8a9f-6016741a47e3)

UI Demo Segmentation: - Good results with 4 foreground points and 4 background points.
![resized_sam_ui_2](https://github.com/facebookresearch/segment-anything/assets/92079088/9eae1a7d-67f4-49d7-8cf2-f45a4e5c1cb6)

My Code Segmentation: - Poor results with the same point placement.
![code_output_2](https://github.com/facebookresearch/segment-anything/assets/92079088/123bc20f-4374-494a-8dd6-4fb79ec3bac2)

I would appreciate any insights into why this discrepancy is happening.
Could it be related to hidden hyperparameter settings, optimizers, or learning rates used in the UI demo that aren't included in the GitHub code?
If this is the case, would it be possible to provide some guidance.

Contributor guide

Open the contributing guide

Research direction

Start by running notebooks/predictor_example.ipynb with the vit_h checkpoint and the point placements shown in the issue. Compare its inputs and outputs with the official UI demo, then trace the notebook's predictor configuration to identify the source of the discrepancy; done means explaining the difference and documenting reproducible matching behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.