Cluster-level resources
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Currently we have worker-level resources, which are useful for limiting access to things that are limited per-worker (memory, GPU, etc...). Sometimes tasks will consume resources that are globally limited though (database connections for example). Right now users have a few options:
- Divide the global resource evenly between workers and set per-worker resources accordingly. This doesn't play well with adaptive scaling, and requires knowing the cluster size to apply the limit.
- Use a `distributed.Semaphore` to limit the active usage of a global resource. This effectively sets a global limit, but doesn't let the scheduler see the concurrency limit. As such, the schedule may allocate more resource-consuming tasks than can currently run, with the excess tasks all taking up an execution thread while waiting on the semaphore to run.
Cluster level resources would let the scheduler take this limit into consideration during scheduling.
Contributor guide
Assessment
This issue has not been assessed yet.