file watcher is very CPU hungry in ^8.3.4

We use tsup with turbo in our monorepo to build ~30 packages. I'm on node 20.18.2, but this issue seems present in node 18 as well.

I recently upgraded from tsup 6.7.0 to 8.3.6 and noticed our local build/watch step (`tsup --watch`) started consuming significantly more CPU (70-100% per process rather than 1-2%), seemingly due to the file watcher. I downgraded back to 6.7.0 and slowly upgraded through each major version, then minor/patch versions, to find out where the issues started.

8.3.0 is the last version where tsup is ~the same CPU usage. Once I bump past 8.3.0, CPU per process spikes and stays high until killed.

Some notes:
- I first tried using `pnpm.overrides` to downgrade esbuild and rollup to specific prior versions, but didn't see any effect.
- 8.1.1 just doesn't work and all builds exit with `Cannot find module '@swc/core'` (fixed in 8.1.2)
- 8.3.1 and 8.3.3 don't appear to be installable from npm, despite it being in releases (`No matching version found for tsup@8.3.3`)

Happy to provide more info and continue debugging if helpful!

Reproduce the regression with `tsup --watch` in the reported monorepo setup, comparing 8.3.0 with later versions on Node 20.18.2 and, if useful, Node 18. Investigate the file-watcher changes between those releases and verify completion by bringing CPU usage near the 8.3.0 baseline without breaking the build; no source file or test path is named.

Written by the indexing model from the issue text.