dask / dask/dask-cloudprovider

Support multiple workers per VM

Open
#173 33 comments 1 reaction 0 assignees View on GitHub
provider/gcp/vm
Dominant language
Python
Stars
147
Forks
119
PR merge metrics
No merged PRs in 30d

Description

It may be more efficient to run many workers on larger VMs as opposed to larger numbers of worker processes on small VMs, presumably to avoid inter-VM communication. This was a suggestion that resulted from this thread on optimizing some dask array workflows: https://github.com/pystatgen/sgkit/issues/390.

@quasiben mentioned that threads per worker can be controlled with `worker_options={"nthreads": 2 }`, but there appears to be no way to run more than one worker on a single VM.

Contributor guide

Open the contributing guide

Research direction

Start by examining the cloud-provider cluster configuration around worker_options and how a VM's worker process is requested. Compare the current single-worker behavior with the desired multiple-workers-per-VM behavior, then verify that the relevant provider configurations can launch and manage more than one worker on the same VM.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cloud, distributed-systems, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.