Considerations for worker metrics in dashboard / performance reports
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Motivated by dask/distributed#4494, I've been thinking a lot about how worker metrics (including GPU memory/utilization) could be represented in the Distributed dashboard / performance reports; this has culminated in a bunch of PRs playing around with dashboard components and the system monitor on my end, but I realize that since most of that work has only included the same ~5 people, it could be good to open wider discussion on this.
Some of the information desired from #4494 is:
- information on a worker metric over the course of an operation
- peak/average value of a metric during an operation
Some questions/thoughts that come to mind:
- *Would time series plots be useful for tracking worker metrics?* My first thought was to have something like the dashboard's System tab for workers, but it would require some progress in Bokeh (bokeh/bokeh#11101) to get working.
- *Where could statistics (mean/max) of a metric be displayed?* My first thought was either as a label / hover tool in a plot of the metric, or in a sortable table the the one currently used in the Workers tab.
- *Should GPU metrics stay separate from other metrics?* Currently, GPU information is only available in the standalone GPU tab added by dask/distributed#4556, but it shouldn't be difficult to add it to something like the Workers table or a hypothetical Workers time series tab. One thing to note here is that since GPU info is collected and displayed conditional on a user having `pynvml` installed, GPU info will show up even if a user isn't working with GPUs.
Interested to hear more opinions on this.
Contributor guide
Assessment
This issue has not been assessed yet.