Add distributed.worker.memory.terminate to the worker keyword arguments
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
From the docs and a quick scan of the distributed codebase, it looks like the config parameter for the fraction of memory at which Dask should terminate a worker (`distributed.worker.memory.terminate`) can’t be set as a keyword argument during worker setup. Instead, it has to be set with `dask.config.set` or as an environment variable.
As far as I can tell from a brief look, I only see this being set here: https://github.com/dask/distributed/blob/7c1b4dfdc1dc9d690ee1731a8fc94f5665620de7/distributed/nanny.py#L145
Unlike the other arguments, which are set based on keyword arguments, if they exist, and then based on the configuration, in the `worker.py`: https://github.com/dask/distributed/blob/83f8febd32d99b58b75ecd3da710dd8a25618867/distributed/worker.py#L530-L535
If this is true, could we allow this behavior by searching within `worker_kwargs` and grabbing the configuration, if it exists, since we have access to it.
https://github.com/dask/distributed/blob/7c1b4dfdc1dc9d690ee1731a8fc94f5665620de7/distributed/nanny.py#L142-L146
From talking with @quasiben , I know there is some discussion around whether expanding worker arguments / the CLI is a good idea versus pushing complexity to configuration. I'll sidestep this for now, but am interested in the discussion.
Contributor guide
Assessment
This issue has not been assessed yet.