Custom mask bugs out

Is there an existing issue for this?

• I checked and did not find my issue in the already reported ones

Describe the bug

I have an issue with custom drawn masks in 5.4.0 on M2 Mac
at some point mask bugs out and no longer shows me handles, I can back off one step using ctrl+z, but when I try to adjust mask again after that it bugs out again, at one point I had to so severe that one mask caused all my existing and new masks bug out the same way

Steps to reproduce

Not sure really, I just draw custom mask, adjust feathering to near 0 and it bugs out randomly
Im using about 20-30 points at max, but it happened even on very simple masks around 10 points

Expected behavior

No response

Logfile | Screenshot | Screencast

on a screenshot I have enabled "editing" mode for mask + showing mask area, you can see no handles and weird area shown

Commit

No response

Where did you obtain darktable from?

darktable.org / GitHub release

darktable version

5.4.0

What OS are you using?

Mac

What is the version of your OS?

15.7.4

Describe your system

M2 MAX; 64GB RAM

Are you using OpenCL GPU in darktable?

Yes

If yes, what is the GPU card and driver?

integrated

Please provide additional context if applicable. You can attach files too, but might need to rename to .txt or .zip

• Are the steps above reproducible with a fresh edit: yes
• Do you use lua scripts?: no

No source file, test, or entry point is named. Start by reproducing the issue in a fresh edit with a custom mask, near-zero feathering, and 10–30 points on the reported macOS setup; done means mask handles remain visible and the mask area renders correctly after adjustment and undo.