facebookresearch / facebookresearch/segment-anything

Fine-tune SAM on challenging cell segmentation images

Open
#456 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
54.9k
Forks
6.4k
PR merge metrics
No merged PRs in 30d

Description

Recently, there was a slew of tutorials and research papers available for ```SAM``` with which you can fine-tune SAM for pretty much any task. But, I was recently trying to fine-tune the decider of SAM on my cell segmentation **(binary classification)** problem and it gave me an error. Most tutorials are made in mind for datasets where the segmentation part can be easily encapsulated in a bounding box but my image can't be done as that. As there are maybe 30 different cells in one image and it is quite difficult to place bounding box around those cells.

The problem is not training SAM as SAM easily trains on my images rather it it is the moment of plotting the predicted masks by the model. I have tried experimenting with a few code and wrote some functions as well but they all fails. That's why, I am turning to the community if they could provide some support/guidance on this issue. It'll greatly help me.

**References**
* https://medium.com/@rekalantar/segment-anything-model-sam-for-medical-image-segmentation-9344ba57f2ca
* https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SAM/Run_inference_with_MedSAM_using_HuggingFace_Transformers.ipynb
* https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SAM/Fine_tune_SAM_(segment_anything)_on_a_custom_dataset.ipynb

**Image in my Dataset (Mask)**
![11](https://github.com/facebookresearch/segment-anything/assets/112309211/987f10d3-b681-432c-b8a7-3ba6e651ef43)
**Image Copyright belongs to me and my team**

Contributor guide

Open the contributing guide

Research direction

Start with the referenced SAM fine-tuning and MedSAM inference notebooks, then reproduce the failure at the predicted-mask plotting step using the multi-cell image described here. The issue does not include a traceback, repository file, or exact expected output, so first isolate those details; done means identifying a reproducible plotting failure and documenting the expected handling of the cell masks.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.