Worse segmentation output on frames with input points

When using `propagate_in_video` I am seeing noticeably worse segmentation on the frames where I have points specified.
For example, if I specify points on frame 5 of a 15 frame stack, the output on frame 5 will look worse than the rest of the stack. If instead I specify points on frame 4, then the output on frame 5 will look as expected, but the output on frame 4 will look worse than the rest of the stack. If I have points on frame 5 and 8, then both will look worse than the rest. See below for a snippet of what I am doing, with or without the back propagation step, I notice the same behavior.

I am using sam2.1_hiera_s.pt checkpoint.

Anything that I am doing wrong of that I could improve? or is it expected behavior?

```
# Add points to each specified frame
for slice_num in points_by_slice:
for obj_id in points_by_slice[slice_num]:
points = np.array(points_by_slice[slice_num][obj_id]["points"])
labels = np.array(points_by_slice[slice_num][obj_id]["labels"])
_, _, _ = self.predictor.add_new_points_or_box(
inference_state=self.inference_state,
frame_idx=slice_num,
obj_id=obj_id,
points=points,
labels=labels
)

# Propagate the segmentation through the entire video
video_segments = {}
for out_frame_idx, out_obj_ids, out_mask_logits in self.predictor.propagate_in_video(self.inference_state):
video_segments[out_frame_idx] = {
out_obj_id: (out_mask_logits[i] > 0.0).cpu().numpy()
for i, out_obj_id in enumerate(out_obj_ids)
}

# Back propagation
for out_frame_idx, out_obj_ids, out_mask_logits in self.predictor.propagate_in_video(self.inference_state,
reverse=True):
video_segments[out_frame_idx] = {
out_obj_id: (out_mask_logits[i] > 0.0).cpu().numpy()
for i, out_obj_id in enumerate(out_obj_ids)
}
self.print_seg(video_segments)
```

Here is an output example with the dog dataset with points on that slice (bad)

and without points on that frames (good)