Workers in a LocalCluster appear to stall at indeterminate time through computation

I am still trying to narrow down the source of my issue, but I am raising the issue now incase someone is able to jump in with some insight that helps me either solve it or find an MRE.

### **What happened**:
I create a LocalCluster with between 32-90 workers (depending on what size server I am running on).

I create and submit a rather large task graph (~3million tasks) composed of entirely Dask Dataframe operations.

At indeterminate times through the computation, some of my workers abruptly drop
to 0-4% CPU usage. They still hold keys in memory, and still have tasks listed
in their processing queue, but they do not make any progress.

If I do not intervene, all other workers exhaust their processing queues (taking
new tasks from the scheduler & completing them) until
the only tasks remaining are those that depend on tasks currently in the queue
of the stalled worker(s). At this point, all workers in the cluster sit idle.

If I manually sigterm the offending worker(s), then the computation is able to finish successfully.

I originally thought I might be seeing an instance of https://github.com/dask/distributed/issues/3761, but now I think my problem is different.

Often (but not always), the stalled workers have a status of `closing`. More
often (almost always, but not 100%), the workers emit a log line stating something like `Stopping
worker at `, which tells me something is invoking the `Worker.close()` method.

I've noticed that in general for every worker that hits this state, there are
~3-4 other workers that close successfully.

After adjusting the connection timeouts
(`distributed.comm.timeouts.tcp/.connect`) to 90s each, I am yet to encounter
this problem again.

I have two problems:
1. I am unable to determine why the workers are being closed. I suspect it is
related to them timing out
2. The workers hang on to tasks, meaning that my compute stalls without human
intervention.

I can live with `1` as long as `2` isn't true, but ideally I'd understand why
`1` was occurring to fix the source of the problem.

### **What you expected to happen**:
The workers to close successfully, or not be closed at all.

### **Minimal Complete Verifiable Example**:

I am yet to create a MRE for this, and am losing hope that I will succeed at doing so.

### **Anything else we need to know?**:

I realise that this bug report might lack enough detail to work with - I am sharing it in the hope that someone may help point me in a direction to dig further. I will post updates as I uncover more.

### **Environment**:

- Dask/Distributed version: 2.18.0 (also encountered on 2.16.0, 2.17.0)
- Python version: 3.7.7
- Operating System: Linux
- Install method (conda, pip, source): Conda