facebookresearch / facebookresearch/sam2

Reset of the inference state/memory upon adding an additional box

Open
#625 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
19.9k
Forks
2.5k
PR merge metrics
No merged PRs in 30d

Description

Hello!

While processing a video and tracking a single object, I wish to reset the inference state and force SAM model to start focusing on different part of the object in later frames.

Currently I have something along the lines of:

```python3
from sam2.build_sam import build_sam2_video_predictor
sam_model = build_sam2_video_predictor(
sam_model_config, os.path.join(working_dir, sam_model_checkpoint), device=device
)
inference_state = sam_model.init_state(video_path=subfolder)
for frame_index in sorted(sequence.frames.keys()): # Making sure that I go from the earliest frame to the latest
# The frame index can be 0, 1, but also 100 e.g. a new grounding can be provided much later
box = sequence.get_box(frame_index, instance_id)
if box is not None:
sam_model.add_new_points_or_box(
obj_id=box.instance_id,
inference_state=inference_state,
box=box.xyxy,
frame_idx=frame_index,
)

for out_frame_index, _, out_mask_logits in sam_model.propagate_in_video(inference_state):
# Collect prediction
```

I tried setting: https://github.com/facebookresearch/sam2/blob/2b90b9f5ceec907a1c18123530e92e794ad901a4/sam2/sam2_video_predictor.py#L29 to True, however, it seems like this function does not exist: https://github.com/search?q=repo%3Afacebookresearch%2Fsam2%20def%20_clear_obj_non_cond_mem_around_input(&type=code, after I have changed the relevant lines in: https://github.com/facebookresearch/sam2/blob/2b90b9f5ceec907a1c18123530e92e794ad901a4/sam2/sam2_video_predictor.py#L598 and https://github.com/facebookresearch/sam2/blob/2b90b9f5ceec907a1c18123530e92e794ad901a4/sam2/sam2_video_predictor.py#L526C30-L526C66 to https://github.com/facebookresearch/sam2/blob/2b90b9f5ceec907a1c18123530e92e794ad901a4/sam2/sam2_video_predictor.py#L956 it did not seem to make a difference.

Thank you for looking into this!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.