dask / dask/distributed

Cluster-level resources

Open
#6,382 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Currently we have worker-level resources, which are useful for limiting access to things that are limited per-worker (memory, GPU, etc...). Sometimes tasks will consume resources that are globally limited though (database connections for example). Right now users have a few options:

- Divide the global resource evenly between workers and set per-worker resources accordingly. This doesn't play well with adaptive scaling, and requires knowing the cluster size to apply the limit.
- Use a `distributed.Semaphore` to limit the active usage of a global resource. This effectively sets a global limit, but doesn't let the scheduler see the concurrency limit. As such, the schedule may allocate more resource-consuming tasks than can currently run, with the excess tasks all taking up an execution thread while waiting on the semaphore to run.

Cluster level resources would let the scheduler take this limit into consideration during scheduling.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.