Tests can't run in parallel because of fixed ports

**What happened**:
Tests take a rather long time to run. So it would be nice to run them in parallel with pytest-xdist. However, they randomly fail to start, because they use the same port, and a second test is unable to open a port while another listens on a port. Thus you sometimes get a test error like:
```
OSError: [Errno 98] Address already in use
```

**What you expected to happen**:
Tests all run in parallel

**Minimal Complete Verifiable Example**:

```
pytest -n auto
```

**Environment**:

- Dask version: 2021.07.2
- Python version: 3.9.6
- Operating System: Fedora 34
- Install method (conda, pip, source): Source