Flaky test survey

Ugh, so I just went through all of the flaky tests in https://dask.org/distributed/test_report.html , finding the failing test, a link to the failure, and a quick summary of my take on things. I was about 90% through when I accidentally closed the tab. Sadly Chrome doesn't seem to have it in memory.

So, rather than a detailed description of everything, we're going to get my summary of a summary.

### High level thoughts

I think that people right now are hoping that there is a silver bullet to make CI solid. I don't think that this is the case. I think that there are a few issues. Some of them are systemic. Some of them are individual tests that are genuinely not solid. I think that we need to take a kitchen sink approach to CI

### CLI tests

In CLI tests, often the tests that fail are relying on a fixed port. This should be safe, but for some reason it isn't. I propose two options:

1. We always use 8786 for all CLI tests, and then if any of them start failing we know that something is up.
2. We get very careful about verifying that servers actually came up

### Windows

In windows tests we often fail due to permissions errors on either sockets or directories

- OSError: [WinError 10013] An attempt was made to access a socket in a way forbidden by its access permissions

https://github.com/dask/distributed/runs/5704828918?check_suite_focus=true

- NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\_dask_test_workercy33vgaa\\dask-worker-space\\worker-s64n_p20.dirlock'

https://github.com/dask/distributed/runs/5645525730?check_suite_focus=true

### Other

I'm also curious about distributed.tests.test_asyncprocess.test_signal , which is part of the codebase that I don't get well, but that fails in a way similar to how lots of Nanny-related tests fail.

- https://github.com/dask/distributed/runs/5747097630?check_suite_focus=true

There's distributed.tests.test_spill.test_weakref_cache[60-SupportsWeakRef-True] , which has been active recently and that I know @crusaderky is working on

test_worker_reconnects_mid_compute and test_worker_reconnects_mid_compute_multiple_states_on_scheduler don't show up in this view, but I see them often on PRs. @fjetter git blame points to you on this one. https://github.com/dask/distributed/runs/5789790148?check_suite_focus=true

### What to do next

There are a variety of small issues (that I had previously painstakingly described). I'm happy to own some of these. I'm probably the best able to navigate them without getting stuck. However, there are some issues, listed above, where I could use help. In particular with Windows, `distributed.tests.test_asyncprocess.test_signal`, and some of the newer tests that came in after I had context (@fjetter I might leave things like `test_worker_reconnects_mid_compute` to you)

I *do not* think that the right thing to do here is to wait and hope that dropping tornado will solve all of these problems. I could definitely see some of these improving, but I don't think that it will be more than half. I'm open to being correct here though.