Reserve memory capacity in worker operations?

I'm just thinking here about what mechanisms we'd need to fix https://github.com/dask/distributed/issues/6208. If we want to avoid fetching more data that we have memory for, we need to know how much space we _do_ have for fetched data. But because there are many systems producing memory, maybe they need some way to cooperate on how much memory each is going to need.

Currently, workers don't take into account how much memory an operation will require:
- size of data fetched from peers
- size of data un-spilled from disk
- estimated output size of a task and the estimated runtime memory usage of a task are not currently tracked (see https://github.com/dask/distributed/issues/5250, https://github.com/dask/distributed/issues/4891), but if they someday were, maybe they'd be on this list too.

Right now, workers transfer and un-spill as much data as they want:
- https://github.com/dask/distributed/issues/6208
- https://github.com/dask/distributed/issues/4424#issuecomment-1058478718

If they start using too much memory, _hopefully_ they pause. (Pausing has a limited effect on actually stopping more transfers or un-spilling, but that will get better https://github.com/dask/distributed/pull/6195 https://github.com/dask/distributed/issues/5900 https://github.com/dask/distributed/issues/5996.) Because pause-checking runs at an interval, also it's possible to get unlucky and use up enough memory to start the OS thrashing, which also prevents the memory monitor from intervening further. But even if pausing works, it's still disruptive and not ideal. Sprinting and stopping all the time is not a good a way to win a race, compared to a steady pace.

Instead of waiting for too much memory to be used, then pausing, maybe we could not start the memory-producing operation until we had some memory capacity "reserved" for its output? A simple data structure similar to a [`CapacityLimiter`](https://trio.readthedocs.io/en/stable/reference-core.html#trio.CapacityLimiter) would work here (with a variable-sized `acquire` and `release`). This would create backpressure and let subsystems cooperate to proactively avoid using too much memory, versus reactively pausing everything when it happens.

Of course, we'd have to be careful to a) not deadlock because of this and b) not thrash with this just like the OS does (if all the operations use up nearly the full `CapacityLimiter`, they will get done eventually, but so slowly that it'll feel like nothing's happening—better to just fail, probably, and say "this can't be done without increasing worker memory"). Note that we face the same problems with pausing.