Alternative `worker_client` usage model
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Currently if users would like to do some custom task submission as part of their task graph, Distributed provides [`worker_client`]( https://distributed.dask.org/en/stable/task-launch.html#connection-with-context-manager ). This does some `secede`/`rejoin` logic to allow the `Worker` to leave the pool and the rejoin later. However this comes with some caveats:
1. A patched copy of [`concurrent.futures`]( https://github.com/dask/distributed/blob/d4adb3a0a8aab4916369cfbf40f0e8be10d8a762/distributed/_concurrent_futures_thread.py )
2. The former requires repatching to get recent `concurrent.futures` updates
3. Lack of visibility to users running code that may rely on this feature (if they want to understand their workflow's usage)
4. Inability to use these workflows with the local scheduler or alternative `Executor`s easily
An alternative approach might be to surface the needed to perform dynamic submission at a higher level (like as part of the graph itself). For example one might decorate functions that would like access to the Client and provide a `client` argument, which the decorator would fill in (as part of task submission). This would make it more visible to users of workflows depending on this functionality. Also the Client and Scheduler could take this into account when orchestrating task submission. This would also avoid the need to think about how workers interact with this feature (since the Client and Scheduler could take care of that in advance).
There may be other approaches that are better than this option. Raising mostly to get the discussion on better alternatives that might simplify maintenance/usage
Contributor guide
Assessment
This issue has not been assessed yet.