Pew-pew plot maybe? cripples cluster if WebSocket disconnects

I had a computation running smoothly for many minutes on a Coiled cluster. I had ~10 JupyterLab panes open showing various parts of the dashboard, including the cluster map (pew-pew).

The computation got to a point where task throughput suddenly increased significantly—many tasks completed per second. I noticed that all the dashboard plots became very laggy, started going blank for many seconds, etc. Since plots such as Worker CPU timeseries stopped advancing in time, it seems like the scheduler was very overwhelmed by something.

I looked at scheduler logs and happened to notice many errors thrown from https://github.com/dask/distributed/blob/8cff8b714cee6396c62707788cdb9899bb2ff330/distributed/diagnostics/websocket.py#L64

Unfortunately I've now lost the logs, but I remember the error was something about the WebSocket being closed.

During brief flashes that the worker memory plot reappeared, I saw that many of the workers were under significant memory pressure, largely due to managed memory. I'm guessing this is related to https://github.com/dask/distributed/issues/5114, where when the scheduler is under load, it can't tell workers to release keys, so they keep piling up memory.

So the theory is:
- an exception from the WebSocket happened at every transition (of which there were many)
- handling this slowed down the scheduler a lot
- workers piled up extra keys because of the hobbled scheduler

But this is just a theory. Does it seem at all plausible?

cc @jacobtomlinson

**Environment**:

- Dask version: `main`-ish
- Python version: 3.9.6
- Operating System: linux (cluster)
- Install method (conda, pip, source): source