Investigate asyncio.wait_for for possible bug

I've encountered situations that I think point to asyncio.wait_for not actually cancelling the underlying coroutine if the timeout is reached. I've had to work around this in a couple of situations:

- https://github.com/dask/distributed/pull/6155
- https://github.com/dask/distributed/pull/6186

See for a possible root cause: https://github.com/python/cpython/issues/86296

We use `asyncio.wait_for` commonly throughout the codebase. If it genuinely sometimes misses then we should find a workaround, including possibly writing this ourselves.

cc @graingert do you have any thoughts on the CPython bug? Could it be causing what we're seeing? Do you see an easy solution here?