HumanSignal / HumanSignal/label-studio-ml-backend

SAM Model Predict Bug

Open
#278 2 comments 3 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.1k
Forks
490
Avg merge
1d 14h
Merged PRs (30d)
3

Description

I had followed https://github.com/heartexlabs/label-studio-ml-backend/tree/master/label_studio_ml/examples/segment_anything_model to deploy **SAM model**

But there are some bugs of **SAM** predictions:
### The coordinates of the mask will be offset from the target I expected, but in fact the shape of the mask is correct

Examples:
1. Here is a 2048*1024 image performance(**Offset**):
![图片](https://github.com/heartexlabs/label-studio-ml-backend/assets/36265886/832fc00a-20d8-4aef-b220-6837b72185d4)

2. Here is a 640*426 image performance(**Correct**):
![图片](https://github.com/heartexlabs/label-studio-ml-backend/assets/36265886/9436ed4c-6056-48c0-a2b6-ece2487f25e6)

3. Here is a 640*480 image performance(**Offset**):
![图片](https://github.com/heartexlabs/label-studio-ml-backend/assets/36265886/79cf8f48-6d1e-49dd-a196-92a64bdc1691)

There are absolutely bugs on the process after prediction, and I think the key of question is the image padding

Here are the original images:
![88a3f9da-berlin_000005_000019_leftImg8bit](https://github.com/heartexlabs/label-studio-ml-backend/assets/36265886/bf96561a-7ebc-4aab-8cb4-c362eeb710ba)
![d6d991d3-000000055150](https://github.com/heartexlabs/label-studio-ml-backend/assets/36265886/de566496-f4ef-4daa-9759-badeddd1e0ca)
![d01284fc-000000167486](https://github.com/heartexlabs/label-studio-ml-backend/assets/36265886/940c416b-a158-4cf8-a0d6-fc904900554f)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in label_studio_ml/examples/segment_anything_model and trace how SAM predictions are processed after inference, focusing on image padding and coordinate conversion. Reproduce the reported cases using the 2048×1024, 640×426, and 640×480 images; done means the predicted masks align with the target images without offsets.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.