`dts` creation impossibly slow.

I've been trying to migrate some project from `unbuild` to `tsup`.
Theres nothing special about it, it's not that big either.

`unbuild` was pretty fast, with 2-3s to chew the whole thing. **dts included**.
I've moved to `tsup`. bundling was amazingly fast, less than 60ms. But `dts` takes 5 seconds.
I do not resolve, it's a plain `dts: true`.

I've looked at the `rollup-plugin-dts` and saw that they the following:

>The plugin works best with already existing .d.ts files generated by the typescript compiler from idiomatic code.

Ok, so lets do that. I run `tsc` with declaration: true, without a declaration map. It takes around 1s and done.
Now tsup takes around 20s to run dts before saying it's out of memory.

I thought there might be some multiple entry points shenanigans, so I tried reducing to only 1 entry point.
dts with no prior `tsc` - 1s. Relatively slow, but fine.
dts with `tsc` - crash after 20s. OOM.
dts with `tsc` _only on the entry file_ - now dts takes 11s but resolves.... with the same exact result as it would if I haven't generated the `d.ts` beforehand.

How can this be? Why would it make things worse??
How can `unbuild` do it so fast with the same exact plugin, less optimized??? (they don't disable anything like tsup does)

I know this is a general typescript issue, but somehow `tsup` turns a slow operation into a full-on odyssey.
Possibly related issues: #840

I tried to work on this on my own, but I couldn't find the source of the issue.