dask / dask/distributed

Add distributed.worker.memory.terminate to the worker keyword arguments

Open
#3,519 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

From the docs and a quick scan of the distributed codebase, it looks like the config parameter for the fraction of memory at which Dask should terminate a worker (`distributed.worker.memory.terminate`) can’t be set as a keyword argument during worker setup. Instead, it has to be set with `dask.config.set` or as an environment variable.

As far as I can tell from a brief look, I only see this being set here: https://github.com/dask/distributed/blob/7c1b4dfdc1dc9d690ee1731a8fc94f5665620de7/distributed/nanny.py#L145

Unlike the other arguments, which are set based on keyword arguments, if they exist, and then based on the configuration, in the `worker.py`: https://github.com/dask/distributed/blob/83f8febd32d99b58b75ecd3da710dd8a25618867/distributed/worker.py#L530-L535

If this is true, could we allow this behavior by searching within `worker_kwargs` and grabbing the configuration, if it exists, since we have access to it.

https://github.com/dask/distributed/blob/7c1b4dfdc1dc9d690ee1731a8fc94f5665620de7/distributed/nanny.py#L142-L146

From talking with @quasiben , I know there is some discussion around whether expanding worker arguments / the CLI is a good idea versus pushing complexity to configuration. I'll sidestep this for now, but am interested in the discussion.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.