Still a problem when label objects on different frames in a video.

Thanks to Sam3, the work has become much easier. However， still have problem I cannot solve by myself.

This issue was raised at https://github.com/facebookresearch/sam2/issues/462
When I tried to use sam2 predictor which is from:

sam3_model = build_sam3_video_model(checkpoint_path=checkpoint_path, bpe_path=bpe_path, device=device)
predictor = sam3_model.tracker

using the add-box or add-point func:

frame_idx_out, out_obj_ids, low_res_masks, video_res_masks = predictor.add_new_points_or_box(
inference_state=inference_state,
frame_idx=pt_frame_idx,
obj_id=plan_obj_id,
points=points_tensor,
labels=points_labels_tensor,
)

I chose 3 different object s on same frame (like 0), everything works fine. however, when I select 3 different object s on different frames(like 1,2,40), the last two objects are lost after "propagate func" :

predictor.propagate_in_video(
inference_state, start_frame_idx=0, max_frame_num_to_track=1300, reverse=False, propagate_preflight=True
)

Do you guys have any idea?

Reproduce the reported case with predictor.add_new_points_or_box on frames 1, 2, and 40, then call predictor.propagate_in_video as shown. Start by tracing those tracker entry points and compare propagation with three objects selected on one frame; done means all selected objects remain available after propagation.

Written by the indexing model from the issue text.