Negative Prompting Class Label Inbalance?

Hi all! Working with SAM2 and trying to refine a prompting pipeline. I wanted to ask if anyone else has had issues with the negative prompts. Specifically, when I prompt the model with a single positive prompt and a single negative prompt, or a single positive prompt, it tend to perform fairly well, but when I saturate the input with multiple negative prompts the output begins to seriously have issues. However, within the demo I did not see this happen, so I believe I must be missing something. If anyone has a suggestion on how to solve this please let me know!