RFC Exponential backoff in connect causing test instability?

An interesting thing I discovered in https://github.com/dask/distributed/pull/5842 connected to test runtimes but also connect timeouts.

The tests there have a typical pattern like

```python
with popen("dask-scheduler"):
with popen("dask-worker"):
with Client("scheduler-address"):
...
```

which spawns the scheduler and a worker using the CLI as dedicated subprocesses. `popen` returns pretty fast but when it returns, the scheduler/worker is not up, yet, and ready to accept requests. The process is probably up but the dask server code is still in startup. I noticed that all of these tests run at least 2-3 seconds even if they do not do anything. I had a brief look into it and noticed that during these ~2-3 seconds, the Client is attempting to connect to the scheduler. However, since it is not up and ready to accept a connection the Client receives a "Connection refused" exception and initiates a retry with exponential backoff (see https://github.com/dask/distributed/blob/f3b0ddb6eeb928d604c0ac13e8b63b511abe3b27/distributed/comm/core.py#L279-L313). If the processes are slow to come up, that's easily a few seconds spent in backoff.

```python
# Once we're beyond ~10 attempts the backoffs *may* be very large.
# These are the *maximum* backoffs / upper_cap. The retry algorithm
# uses the below values and samples the actual backoff using ` random.uniform(0, upper_cap)`
In [1]: [0.01 * (2**attempt) for attempt in range(0, 12)]
Out[1]: [0.01, 0.02, 0.04, 0.08, 0.16, 0.32, 0.64, 1.28, 2.56, 5.12, 10.24, 20.48]
```

This primarily impacts our CLI tests which use dedicated subprocesses and wait for everything to come up and connect autonomously. Most of our tests using `utils_test.cluster` or `utils_test.gen_cluster` synchronize startup such that workers and the client are only started once the scheduler is up, such that we do not face this problem in most situations. There are a few tests outside of the CLI tests which are using a similar pattern, though, e.g. `test_variable.py::test_variable_in_task` or `test_client.py::test_reconnect`

I'm wondering
* Are there other patterns where we might run into this connect issue
* I have a strong opinion that we need a retry with backoff in production. I'm not so sure about our test suite and am wondering if disabling it for CI is something to consider

cc @crusaderky @graingert

----

Edit: Triggered a branch with hard coded instead of exponential backoff, see https://github.com/fjetter/distributed/actions/runs/1881687117