Mask identity jumping between similar objects

Hi all,

I’m encountering an issue when using SAM2 for multi-object video segmentation/tracking, and I’m wondering whether others have seen similar behavior or have suggestions to mitigate it.

**Setup**

**Task**: Generate masklets for fruit flies in a video

**Method**:

1. Provide an initial mask prompt on the first frame
2. Use SAM2 to propagate masks across the video

**Objects**: Multiple fruit flies with very similar appearance

**Observed issue**

The predicted mask for a given object occasionally jumps to a different fly, even when:

- The two flies are not spatially close
- There is no obvious occlusion or interaction
- Once the jump happens, the identity is effectively lost and continues tracking the wrong fly

I’ve attached a short video demonstrating the issue. In the video, the initial mask is correct, but at certain frames the mask abruptly switches to another fly elsewhere in the scene.

https://github.com/user-attachments/assets/ed7afc3c-3a24-4189-b4ab-42449605ddd6

Since the flies have very similar appearance, I suspect this may be an identity ambiguity issue in the model, but I’m wondering whether others have observed similar behavior and if there are any known workarounds or best practices to reduce it. Otherwise, I’m curious if this is a known limitation of the current model design.