dask / dask/distributed

Considerations for worker metrics in dashboard / performance reports

Open
#4,668 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Motivated by dask/distributed#4494, I've been thinking a lot about how worker metrics (including GPU memory/utilization) could be represented in the Distributed dashboard / performance reports; this has culminated in a bunch of PRs playing around with dashboard components and the system monitor on my end, but I realize that since most of that work has only included the same ~5 people, it could be good to open wider discussion on this.

Some of the information desired from #4494 is:

- information on a worker metric over the course of an operation
- peak/average value of a metric during an operation

Some questions/thoughts that come to mind:

- *Would time series plots be useful for tracking worker metrics?* My first thought was to have something like the dashboard's System tab for workers, but it would require some progress in Bokeh (bokeh/bokeh#11101) to get working.
- *Where could statistics (mean/max) of a metric be displayed?* My first thought was either as a label / hover tool in a plot of the metric, or in a sortable table the the one currently used in the Workers tab.
- *Should GPU metrics stay separate from other metrics?* Currently, GPU information is only available in the standalone GPU tab added by dask/distributed#4556, but it shouldn't be difficult to add it to something like the Workers table or a hypothetical Workers time series tab. One thing to note here is that since GPU info is collected and displayed conditional on a user having `pynvml` installed, GPU info will show up even if a user isn't working with GPUs.

Interested to hear more opinions on this.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.