Bug when objects with different `frame_idx` are added using `predictor.add_new_points_or_box`

I have different object entering the FoV of my camera at different frame index. In order to track all these objects I do the following:

```py
for fidx, obj_id, box_coords in zip(ann_frame_idxs, ann_obj_ids, boxes):
_, out_ids, out_logits = predictor.add_new_points_or_box(
inference_state=inference_state,
frame_idx=fidx,
obj_id=obj_id,
box=box_coords,
)
```

When all the boxes added have the same `frame_idx`, in this case 0. Everything seems to work as expected:

Frame0
Frame1

However, when boxes with different `frame_idx` are added, `EdgeTAM` does not behave as expected. Deleting all the previous tracks:

Frame0
Frame1