[CI] Warnings about used ports

Lately we are observing hundreds of warnings about the dashboard ports being occupied. While I understand that we occasionally may have an overlap, I would expect every test to close the HTTP server properly such that these overlaps happen only infrequently.
I can't help but wonder if this connects to our recent CI instabilities and/or CI performance.

Example https://github.com/dask/distributed/pull/4805/checks?check_run_id=2556648953

```
distributed/tests/test_multi_locks.py::test_multiple_locks
/home/runner/work/distributed/distributed/distributed/node.py:160: UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 46611 instead
warnings.warn(

distributed/tests/test_multi_locks.py::test_num_locks
/home/runner/work/distributed/distributed/distributed/node.py:160: UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 46079 instead
warnings.warn(

distributed/tests/test_nanny.py::test_lifetime
/home/runner/work/distributed/distributed/distributed/node.py:160: UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 37239 instead
warnings.warn(

distributed/tests/test_nanny.py::test_nanny_closes_cleanly

```