dask / dask/distributed

Worker count prometheus metric should expose disjoint counters

Open
#7,461 3 comments 0 reactions 0 assignees View on GitHub
diagnostics good second issue
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

The current prometheus metric that exposes number workers known to the scheduler is ill defined.

https://github.com/dask/distributed/blob/3e793f7283f575a73d754da3e313259466cbee33/distributed/http/scheduler/prometheus/core.py#L35-L42

It's current definition is something like

`connected` is the number of all connected workers
`idle/saturated` is the number of workers in the respective state

but these sets are not disjoint. Saturated and idle are disjoint and both are subsets of connected.
Ideally, this metric would expose gauges s.t. summing all labels would yield the total number of workers similar to how we expose task states

https://github.com/dask/distributed/blob/3e793f7283f575a73d754da3e313259466cbee33/distributed/http/scheduler/prometheus/core.py#L76-L78

The implementation should ideally not iterate over all workers every time the metrics are collected but instead the scheduler should maintain the count online.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.