Negative point prompts are being ignored

I am trying to create masks on satellite/aerial imagery and often times I need to refine masks with multiple negative point prompts. Typically I use 2 or so positive prompts and then 4 or 5 negative prompts.

Curiously, I have found that SAM (1) is much better then SAM2 at scenarios where there is negative prompting required.

At times, SAM2 does not seem to even take negative point prompts into account when forming masks.

Why might this be? Is this user error? Is there a config that I should change? Thanks.