facebookresearch / facebookresearch/sam3

BUG? Bbox prompt does not work with prompted text "visual"?

Open
#263 3 comments 4 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
11.7k
Forks
1.8k
PR merge metrics
No merged PRs in 30d

Description

I prompted SAM 3 model in the first frame with the bounding box of the person and the prompted text was set to the default option "visual". However, the model did not produce output for the rest of the frames, except the first one, saying "no objects detected".

response = predictor.handle_request(
request=dict(
type="add_prompt",
session_id=session_id,
frame_index=0,
text="visual",
bounding_boxes = box_abs_to_rel_coords(H, W, 382, 163, 486, 324),
bounding_box_labels = [1]
)
)

outputs_per_frame = propagate_in_video(predictor, session_id) # this did not find any object

What is the issue?
Is it the fact that the text prompt should always be consistent with the bbox? But, in this case, why would you have an available option "visual" for the cases of only the box prompts?

![Image](https://github.com/user-attachments/assets/a4b46651-a391-4599-94ac-1779786d4465)
![Image](https://github.com/user-attachments/assets/3e666edd-ee01-471b-9a50-3a3152bdff88)
![Image](https://github.com/user-attachments/assets/b5737be9-f7fe-4c15-9902-dc28de35f71b)
![Image](https://github.com/user-attachments/assets/dcb44fc8-efd1-4b33-b9ef-de9bcd6f4825)

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the request using the shown add_prompt call with text="visual" and the bounding box, then run propagate_in_video for the session. Read the prompt-handling and propagation entry points involved in these calls and compare the first-frame result with later-frame detection. Done means the prompted person is detected and propagated across the remaining frames, or the incompatibility is clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.