Forward `multiprocessing.current_process().authkey` to workers
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
I'm using a [`multiprocessing` Manager](https://docs.python.org/3/library/multiprocessing.html#managers) to enable communication between the main process and it's workers (for progress reporting). I also want to support `dask_jobqueue.SLURMCluster` as a method to distribute work. My problem is the following:
To connect to the manager in the main process, the client needs an `authkey`. However, [When a proxy is pickled the authkey is deliberately dropped](https://bugs.python.org/msg214582), due to security concerns.
Python's "regular" multiprocessing makes sure that `multiprocessing.current_process().authkey` is propagated to spawned or forked processes, so that they are still able to communicate with the main process.
With `dask_jobqueue.SLURMCluster`, however, this requires some manual setup to get the main process' `authkey` to the workers.
I'm suggesting to implement a secure way to propagate `multiprocessing.current_process().authkey` to the worker.
Related issue: https://github.com/python/cpython/issues/139801
Contributor guide
Assessment
This issue has not been assessed yet.