dask / dask/distributed

Forward `multiprocessing.current_process().authkey` to workers

Open
#9,122 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I'm using a [`multiprocessing` Manager](https://docs.python.org/3/library/multiprocessing.html#managers) to enable communication between the main process and it's workers (for progress reporting). I also want to support `dask_jobqueue.SLURMCluster` as a method to distribute work. My problem is the following:

To connect to the manager in the main process, the client needs an `authkey`. However, [When a proxy is pickled the authkey is deliberately dropped](https://bugs.python.org/msg214582), due to security concerns.
Python's "regular" multiprocessing makes sure that `multiprocessing.current_process().authkey` is propagated to spawned or forked processes, so that they are still able to communicate with the main process.

With `dask_jobqueue.SLURMCluster`, however, this requires some manual setup to get the main process' `authkey` to the workers.

I'm suggesting to implement a secure way to propagate `multiprocessing.current_process().authkey` to the worker.

Related issue: https://github.com/python/cpython/issues/139801

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.