Request for documentation on tuning for scaling to thousand-node cluster

Hello,

I have been trying to connect my client to a 3000-worker cluster across 62 nodes (powerful instances- each with 768GB RAM).
I've been running into issues - tried several days tuning various things - to no avail.

Dask promises to scale to thousand-node clusters ("[Dask's schedulers scale to thousand-node clusters](https://dask.org/)"). yet I am a bit in the dark as to what exactly I need to do to achieve this.

It would be great if there was a clear documentation or guideline as to how to best tune parameters. The difficulty is that there are multiple layers - tcp layer, dask distributed configuration, cluster specific configuration, etc.

(1) I spent a day tuning kernel parameters on my linux scheduler/client machines, such as queue lengths, FD limits, etc.
(2) I tried various combinations of [dask Configuration ](https://docs.dask.org/en/latest/configuration-reference.html).
(3) I'm also using Dask-yarn on top; so I tried tuning Dask-yarn and Hadoop settings.

All of these efforts have not been successful - I think that a documentation for scaling to thousand-node cluster from Dask folks would have saved some time, and may have resulted in some success.

Thank you so much!