dask / dask/distributed

DOC: Configuring local_directory/temporary-directory to point to fast storage medium

Open
#4,000 0 comments 0 reactions 0 assignees View on GitHub
documentation
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

A common issue for users deploying on clusters is configuring scratch space to use on `Worker`s. If not configured, it defaults to current working directory (often the home directory), which clusters (or even some times individual machines) will have configured to use NFS. This can be quite slow and result in erratic/bad behavior by the Dask cluster. While it is difficult for us to pick a better default for users, as the right choice may be hard to make in general, we can provide users some guidance on how to configure the Dask cluster and warn them about common pitfalls (NFS-backed home directories, `/tmp` using [tmpfs (in-memory storage)]( https://en.wikipedia.org/wiki/Tmpfs ), etc.). Usually this is just a matter of setting `temporary-directory` in `dask.yaml` or specifying `local_directory` in the `Client` or `Cluster`. It would be good to lay this out in our docs somewhere to make it easy to refer to for users starting out or setting back up in a new environment.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.