facebookresearch / facebookresearch/sam2
Reset of the inference state/memory upon adding an additional box
- Dominant language
- Jupyter Notebook
- Stars
- 19.9k
- Forks
- 2.5k
- PR merge metrics
- No merged PRs in 30d
Description
Hello!
While processing a video and tracking a single object, I wish to reset the inference state and force SAM model to start focusing on different part of the object in later frames.
Currently I have something along the lines of:
```python3
from sam2.build_sam import build_sam2_video_predictor
sam_model = build_sam2_video_predictor(
sam_model_config, os.path.join(working_dir, sam_model_checkpoint), device=device
)
inference_state = sam_model.init_state(video_path=subfolder)
for frame_index in sorted(sequence.frames.keys()): # Making sure that I go from the earliest frame to the latest
# The frame index can be 0, 1, but also 100 e.g. a new grounding can be provided much later
box = sequence.get_box(frame_index, instance_id)
if box is not None:
sam_model.add_new_points_or_box(
obj_id=box.instance_id,
inference_state=inference_state,
box=box.xyxy,
frame_idx=frame_index,
)
for out_frame_index, _, out_mask_logits in sam_model.propagate_in_video(inference_state):
# Collect prediction
```
I tried setting: https://github.com/facebookresearch/sam2/blob/2b90b9f5ceec907a1c18123530e92e794ad901a4/sam2/sam2_video_predictor.py#L29 to True, however, it seems like this function does not exist: https://github.com/search?q=repo%3Afacebookresearch%2Fsam2%20def%20_clear_obj_non_cond_mem_around_input(&type=code, after I have changed the relevant lines in: https://github.com/facebookresearch/sam2/blob/2b90b9f5ceec907a1c18123530e92e794ad901a4/sam2/sam2_video_predictor.py#L598 and https://github.com/facebookresearch/sam2/blob/2b90b9f5ceec907a1c18123530e92e794ad901a4/sam2/sam2_video_predictor.py#L526C30-L526C66 to https://github.com/facebookresearch/sam2/blob/2b90b9f5ceec907a1c18123530e92e794ad901a4/sam2/sam2_video_predictor.py#L956 it did not seem to make a difference.
Thank you for looking into this!
Contributor guide
Assessment
This issue has not been assessed yet.