facebookresearch / facebookresearch/segment-anything

Is it possible to run ONNX decoder with multiple boxes?

Open
#308 1 comment 3 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
54.9k
Forks
6.4k
PR merge metrics
No merged PRs in 30d

Description

I am experimenting with onnx decoder trying to run it with multiple boxes as:
```
point = index * 2;
point_coords[0, point, 0] = rect.Left
point_coords[0, point, 1] = rect.Top
point_coords[0, point + 1, 0] = rect.Right
point_coords[0, point + 1, 1] = rect.Bottom
point_labels[0, point] = 2
point_labels[0, point + 1] = 3
```
where **index** is index of a box, **rect** is the box by index in "post transform to longest side" space **point_coords** and **point_labels** are tensors which will be passed to inference session.
when running inference with single box everything works fine and generated masks are good, trying to add at least 1 additional box leads to masks being mostly black with some white artifacts in it.

Do ONNX decoder supports multi-box input at the moment?
If it does how do I need to put labels on boxes properly?

Contributor guide

Open the contributing guide

Research direction

Review the ONNX decoder entry point and its point_coords and point_labels inputs, then compare the tensor shapes and label values used for one box with the multi-box construction shown. Confirm whether multiple boxes are supported and document the required encoding, or record that the decoder accepts only one box.

Written by the indexing model from the issue text.

Assessment

Domain
computer-vision, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.