dask / dask/distributed

Fine performance metrics: Meter currently-executing tasks

Open
#7,677 0 comments 0 reactions 0 assignees View on GitHub
diagnostics
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

- Part of #7665
- Blocked by #7666

When looking at a real-time plot in Grafana or similar tools, it's useful to know what the cluster is *currently* doing.

In the Worker heartbeat, add a bit to the metrics, in addition to those collected from `Worker.digests_total_new`, that is the coarse time of the tasks that are *currently* executing, as of the moment the heartbeat was fired, and record it under the label `("execute", , "currently-running", "seconds")`.

When a task finishes, subtract all time logged so far for the task (which may be less than the total task runtime; you'll need to keep track of the timestamp of the latest heartbeat). Whenever the worker is idle, "currently-running" must be zero by construction.

Note that, unlike all other metrics in `Worker.digests_total`, this is not monotonically increasing; in Prometheus terms, it's a Gauge, not a Counter.

This ticket solves a problem in Prometheus metrics where the cluster utilization plot over time looks "spikey" whenever you have tasks that start before a scrape point and finish after it. On Coiled, this means any task that lasts longer than 5s (the Prometheus scraping interval).

Of course, this fix is perfect only as long as you do not display the detail of the activity. Once you break down by activity (task-cpu, etc.) in Grafana you will see, for all running tasks, a large positive spike in the actual activities (task-cpu, etc.) matching the task that just finished, which may send the plot well above the number of threads on the cluster, and a matching negative spike in `currently-executing`.
This ticket does not expect to make the by-activity grafana plot nicer than this.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.