Worker count prometheus metric should expose disjoint counters
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
The current prometheus metric that exposes number workers known to the scheduler is ill defined.
https://github.com/dask/distributed/blob/3e793f7283f575a73d754da3e313259466cbee33/distributed/http/scheduler/prometheus/core.py#L35-L42
It's current definition is something like
`connected` is the number of all connected workers
`idle/saturated` is the number of workers in the respective state
but these sets are not disjoint. Saturated and idle are disjoint and both are subsets of connected.
Ideally, this metric would expose gauges s.t. summing all labels would yield the total number of workers similar to how we expose task states
https://github.com/dask/distributed/blob/3e793f7283f575a73d754da3e313259466cbee33/distributed/http/scheduler/prometheus/core.py#L76-L78
The implementation should ideally not iterate over all workers every time the metrics are collected but instead the scheduler should maintain the count online.
Contributor guide
Assessment
This issue has not been assessed yet.