HumanSignal / HumanSignal/label-studio-ml-backend
SAM pre-cache embeddings for entire project
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 490
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 3
Description
Given that the most time-consuming part of using the (excellent) Segment Anything integration is the initial embedding, it would be extremely useful if all images in the project could have their embeddings generated and stored for future use. There is already a TODO here: https://github.com/HumanSignal/label-studio-ml-backend/blob/0dd3d6cbe0f288620156b1a3c05c3ad93fc8fa9b/label_studio_ml/examples/segment_anything_model/sam_predictor.py#L25 but I wanted to highlight how useful this would be.
As it stands, using the SAM model, embeddings are generated via a call to `set_image` in the `predict_sam` function. Presumably these would need to be returned as in `payload = self.set_image(img_path, calculate_embeddings=True)` in the `predict_onnx` function.
The difficulty comes in that the plugin does not seem to have a way to cycle over all images in the project (say, overnight before manual work begins) and cache these results. If I understand correctly, the current cache is stored on the `SAMPredictor` object and a new one of these is created each time an image is loaded (correct me if I'm wrong there).
This would be a very nice addition and if the author isn't working on this already, it would be interesting to hear suggested ways to implement it.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.