dask / dask/distributed

Fine performance metrics meta-issue

Open
#7,665 2 comments 0 reactions 0 assignees View on GitHub
diagnostics
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

#### XREFs
- #7217
- #7565
- #7601
- #7586

In #7586, we started collecting very granular metrics on how workers are spending their time.
Demo: https://gist.github.com/crusaderky/a97f870c51260e63a1c14c20b762f666

As of that PR, we collect metrics in `Worker.digests_total` about:
- `Worker.execute`, broken down by task prefix and activity, with special treatment for failed and cancelled tasks
- `Worker.gather_dep`, broken down by activity, with special treatment for failed and cancelled transfers
- `Worker.get_data`, broken down by activity
- `WorkerMemoryMonitor._spill`, broken down by activity

This issue is a meta-tracker of all potential follow-ups, as well as a place to discuss high level design and cost/benefit ratios holistically.

The follow-ups can be broken down into two high level threads:

## Improve quality and usability of collected data
- #7666
- dask/dask#10084
- #7677
- #7938
- #7671
- #7672
- #7675
- #7676
- #7678
- #7681

## What we do with the data
- #7667
- #7668
- #7679
- #7831
- #7832
- #7848
- #7875
- #7893
- #7908
- #7910
- #7911
- #7680
- #7776
- #7787
- #7825
- #7790
- Other use cases of overlapping computations?

## Finishing touches
- #7673
- #7674

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.