HumanSignal / HumanSignal/label-studio-ml-backend

SAM pre-cache embeddings for entire project

Open
#326 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.1k
Forks
490
Avg merge
1d 14h
Merged PRs (30d)
3

Description

Given that the most time-consuming part of using the (excellent) Segment Anything integration is the initial embedding, it would be extremely useful if all images in the project could have their embeddings generated and stored for future use. There is already a TODO here: https://github.com/HumanSignal/label-studio-ml-backend/blob/0dd3d6cbe0f288620156b1a3c05c3ad93fc8fa9b/label_studio_ml/examples/segment_anything_model/sam_predictor.py#L25 but I wanted to highlight how useful this would be.

As it stands, using the SAM model, embeddings are generated via a call to `set_image` in the `predict_sam` function. Presumably these would need to be returned as in `payload = self.set_image(img_path, calculate_embeddings=True)` in the `predict_onnx` function.

The difficulty comes in that the plugin does not seem to have a way to cycle over all images in the project (say, overnight before manual work begins) and cache these results. If I understand correctly, the current cache is stored on the `SAMPredictor` object and a new one of these is created each time an image is loaded (correct me if I'm wrong there).

This would be a very nice addition and if the author isn't working on this already, it would be interesting to hear suggested ways to implement it.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.