dask / dask/distributed

Add `preload` and `preload_argv` arguments to `Client.__init__`

Open
#5,819 0 comments 0 reactions 0 assignees View on GitHub
enhancement good second issue
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

PR #5773 adds the ability for the `Client` to run preload code, similar to the `Worker` and `Scheduler`. However, @jrbourbeau pointed out that users may be passing `preload` and `preload_argv` to the `Client.__init__` with the intention of passing them to the worker configuration of `LocalCluster`.

As a temporary measure, #5773 only adds `Client` preload code that is specified in the Dask YAML configuration file. Since a similar problem with keyword arguments passthrough will happen if/when plugins are added for the Client, @jrbourbeau proposed deferring adding the `preload` and `preload_argv` keywords. When those are added, there will need to be a deprecation cycle to change how users pass kwargs to the worker configuration through `LocalCluster`. This deprecation cycle would also remove the similar problem that will occur when/if plugins are added.

One possible solution to this is to modify the signature of `LocalCluster.__init__` so that the `**worker_kwargs` that are collected at the end of the constructor become a keyword argument of their own, similar to the existing `scheduler_kwargs`. See https://github.com/dask/distributed/blob/8d0df89be3a33875513199152d23bed220dc91cb/distributed/deploy/local.py#L121-L123

This should not affect the `n_workers` and `threads_per_worker` kwargs, which would still be passed through unmodified.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.