DOC: Configuring local_directory/temporary-directory to point to fast storage medium
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
A common issue for users deploying on clusters is configuring scratch space to use on `Worker`s. If not configured, it defaults to current working directory (often the home directory), which clusters (or even some times individual machines) will have configured to use NFS. This can be quite slow and result in erratic/bad behavior by the Dask cluster. While it is difficult for us to pick a better default for users, as the right choice may be hard to make in general, we can provide users some guidance on how to configure the Dask cluster and warn them about common pitfalls (NFS-backed home directories, `/tmp` using [tmpfs (in-memory storage)]( https://en.wikipedia.org/wiki/Tmpfs ), etc.). Usually this is just a matter of setting `temporary-directory` in `dask.yaml` or specifying `local_directory` in the `Client` or `Cluster`. It would be good to lay this out in our docs somewhere to make it easy to refer to for users starting out or setting back up in a new environment.
Contributor guide
Assessment
This issue has not been assessed yet.