dask / dask/dask-cloudprovider
Support multiple workers per VM
- Dominant language
- Python
- Stars
- 147
- Forks
- 119
- PR merge metrics
- No merged PRs in 30d
Description
It may be more efficient to run many workers on larger VMs as opposed to larger numbers of worker processes on small VMs, presumably to avoid inter-VM communication. This was a suggestion that resulted from this thread on optimizing some dask array workflows: https://github.com/pystatgen/sgkit/issues/390.
@quasiben mentioned that threads per worker can be controlled with `worker_options={"nthreads": 2 }`, but there appears to be no way to run more than one worker on a single VM.
Contributor guide
Research direction
Start by examining the cloud-provider cluster configuration around worker_options and how a VM's worker process is requested. Compare the current single-worker behavior with the desired multiple-workers-per-VM behavior, then verify that the relevant provider configurations can launch and manage more than one worker on the same VM.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cloud, distributed-systems, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100