Fine performance metrics meta-issue
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
#### XREFs
- #7217
- #7565
- #7601
- #7586
In #7586, we started collecting very granular metrics on how workers are spending their time.
Demo: https://gist.github.com/crusaderky/a97f870c51260e63a1c14c20b762f666
As of that PR, we collect metrics in `Worker.digests_total` about:
- `Worker.execute`, broken down by task prefix and activity, with special treatment for failed and cancelled tasks
- `Worker.gather_dep`, broken down by activity, with special treatment for failed and cancelled transfers
- `Worker.get_data`, broken down by activity
- `WorkerMemoryMonitor._spill`, broken down by activity
This issue is a meta-tracker of all potential follow-ups, as well as a place to discuss high level design and cost/benefit ratios holistically.
The follow-ups can be broken down into two high level threads:
## Improve quality and usability of collected data
- #7666
- dask/dask#10084
- #7677
- #7938
- #7671
- #7672
- #7675
- #7676
- #7678
- #7681
## What we do with the data
- #7667
- #7668
- #7679
- #7831
- #7832
- #7848
- #7875
- #7893
- #7908
- #7910
- #7911
- #7680
- #7776
- #7787
- #7825
- #7790
- Other use cases of overlapping computations?
## Finishing touches
- #7673
- #7674
Contributor guide
Assessment
This issue has not been assessed yet.