developmentseed / developmentseed/segment-anything-services

Support customizable prompts, multiple points and bbox for decoder

Open
#7 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
96
Forks
10
PR merge metrics
No merged PRs in 30d

Description

with the current service, you can only supply one point. But SAM supports supplying N points, or a bbox, which should greatly improve mask detection accuracy.

We can support more prompts by adapting `handler_decode.py` to accept a different payload and pass the prompt inputs more flexibly to ort.InferenceSession. This would involve

1. changing the structure of the expected payload. currently handler_decode.py expects

```
{
"image_embeddings": encoded_embedding_string,
"image_shape": img_shape,
"input_label": input_label, # needs to be updated and made more flexible
"input_point": input_point # needs to be updated and made more flexible
}
```

2. Changing how we pass this information to decoding inference. I think we might just be able to pass the same input shapes for two parameters, since the empty dimension I'm adding in both cases probably represents the potential for multiple points.
3. Handle a bbox input, not sure how to set this up yet but can look into it when it becomes a priority. I think handling N points is higher priority since it's easier on the user.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.