dask / dask/distributed

Alternative `worker_client` usage model

Open
#7,311 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Currently if users would like to do some custom task submission as part of their task graph, Distributed provides [`worker_client`]( https://distributed.dask.org/en/stable/task-launch.html#connection-with-context-manager ). This does some `secede`/`rejoin` logic to allow the `Worker` to leave the pool and the rejoin later. However this comes with some caveats:

1. A patched copy of [`concurrent.futures`]( https://github.com/dask/distributed/blob/d4adb3a0a8aab4916369cfbf40f0e8be10d8a762/distributed/_concurrent_futures_thread.py )
2. The former requires repatching to get recent `concurrent.futures` updates
3. Lack of visibility to users running code that may rely on this feature (if they want to understand their workflow's usage)
4. Inability to use these workflows with the local scheduler or alternative `Executor`s easily

An alternative approach might be to surface the needed to perform dynamic submission at a higher level (like as part of the graph itself). For example one might decorate functions that would like access to the Client and provide a `client` argument, which the decorator would fill in (as part of task submission). This would make it more visible to users of workflows depending on this functionality. Also the Client and Scheduler could take this into account when orchestrating task submission. This would also avoid the need to think about how workers interact with this feature (since the Client and Scheduler could take care of that in advance).

There may be other approaches that are better than this option. Raising mostly to get the discussion on better alternatives that might simplify maintenance/usage

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.