facebookresearch / facebookresearch/sam3
Point mode with video predictor, no text prompt
- Dominant language
- Python
- Stars
- 11.7k
- Forks
- 1.8k
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
I installed, adapted and ran the [sam3_video_predictor_example.ipynb] fine as a Python script locally.
I wish to run the predictor initialized with point prompts only (not object/text prompts),
I tried to first (as in the example):
response = predictor.handle_request(
request=dict(
type="start_session",
resource_path=video_path,
)
)
session_id = response["session_id"]
and then removed the first prediction with the text prompt string, and tried to start with point selection as per the latter part of the example:
sample_img = Image.fromarray(load_frame(video_frames_for_vis[0]))
IMG_WIDTH, IMG_HEIGHT = sample_img.size
frame_idx = 0
obj_id = 1 # Should I specify it at all, given no prior prompting/object identification took place?
points_abs = np.array(
[
[760, 550], # positive click
]
)
labels = np.array([1])
abs_to_rel_coords(points_abs, IMG_WIDTH, IMG_HEIGHT, coord_type="point"),
dtype=torch.float32,
)
points_labels_tensor = torch.tensor(labels, dtype=torch.int32)
response = predictor.handle_request(
request=dict(
type="add_prompt",
session_id=session_id,
frame_index=frame_idx,
points=points_tensor,
point_labels=points_labels_tensor,
obj_id=obj_id,
)
)
BUT, at the predictor.handle_request stage I get the error:
....
....
AssertionError: No cached outputs found. Ensure normal propagation has run first to populate the cache.
So my question is - how can I avoid the initial 'text based prompt object selection' on the video?
Contributor guide
Research direction
Start with sam3_video_predictor_example.ipynb and trace the start_session and add_prompt calls in the video predictor flow. Reproduce the point-only sequence that raises “No cached outputs found,” then determine the expected initialization path for point prompts. Done means a session can begin with point prompts without a text-based object selection or cache assertion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python, pytorch
- Domain
- computer-vision, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100