Add `preload` and `preload_argv` arguments to `Client.__init__`
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
PR #5773 adds the ability for the `Client` to run preload code, similar to the `Worker` and `Scheduler`. However, @jrbourbeau pointed out that users may be passing `preload` and `preload_argv` to the `Client.__init__` with the intention of passing them to the worker configuration of `LocalCluster`.
As a temporary measure, #5773 only adds `Client` preload code that is specified in the Dask YAML configuration file. Since a similar problem with keyword arguments passthrough will happen if/when plugins are added for the Client, @jrbourbeau proposed deferring adding the `preload` and `preload_argv` keywords. When those are added, there will need to be a deprecation cycle to change how users pass kwargs to the worker configuration through `LocalCluster`. This deprecation cycle would also remove the similar problem that will occur when/if plugins are added.
One possible solution to this is to modify the signature of `LocalCluster.__init__` so that the `**worker_kwargs` that are collected at the end of the constructor become a keyword argument of their own, similar to the existing `scheduler_kwargs`. See https://github.com/dask/distributed/blob/8d0df89be3a33875513199152d23bed220dc91cb/distributed/deploy/local.py#L121-L123
This should not affect the `n_workers` and `threads_per_worker` kwargs, which would still be passed through unmodified.
Contributor guide
Assessment
This issue has not been assessed yet.