facebookresearch / facebookresearch/sam3

Point mode with video predictor, no text prompt

Open
#368 13 comments 6 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
11.7k
Forks
1.8k
PR merge metrics
No merged PRs in 30d

Description

Hi,
I installed, adapted and ran the [sam3_video_predictor_example.ipynb] fine as a Python script locally.
I wish to run the predictor initialized with point prompts only (not object/text prompts),
I tried to first (as in the example):
response = predictor.handle_request(
request=dict(
type="start_session",
resource_path=video_path,
)
)
session_id = response["session_id"]
and then removed the first prediction with the text prompt string, and tried to start with point selection as per the latter part of the example:
sample_img = Image.fromarray(load_frame(video_frames_for_vis[0]))
IMG_WIDTH, IMG_HEIGHT = sample_img.size
frame_idx = 0
obj_id = 1 # Should I specify it at all, given no prior prompting/object identification took place?
points_abs = np.array(
[
[760, 550], # positive click
]
)
labels = np.array([1])
abs_to_rel_coords(points_abs, IMG_WIDTH, IMG_HEIGHT, coord_type="point"),
dtype=torch.float32,
)
points_labels_tensor = torch.tensor(labels, dtype=torch.int32)

response = predictor.handle_request(
request=dict(
type="add_prompt",
session_id=session_id,
frame_index=frame_idx,
points=points_tensor,
point_labels=points_labels_tensor,
obj_id=obj_id,
)
)

BUT, at the predictor.handle_request stage I get the error:
....
....
AssertionError: No cached outputs found. Ensure normal propagation has run first to populate the cache.

So my question is - how can I avoid the initial 'text based prompt object selection' on the video?

Contributor guide

Open the contributing guide

Research direction

Start with sam3_video_predictor_example.ipynb and trace the start_session and add_prompt calls in the video predictor flow. Reproduce the point-only sequence that raises “No cached outputs found,” then determine the expected initialization path for point prompts. Done means a session can begin with point prompts without a text-based object selection or cache assertion.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python, pytorch
Domain
computer-vision, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.