Fine performance metrics: Meter task re-execution after losing a worker
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
- Part of #7665
- Complements #7678
When a worker dies and you lose tasks in `memory` state, they transition back to `released` on the scheduler and are re-computed somewhere else.
We would like to know how much time we spent re-computing tasks after a worker dies. This could inform the user e.g. to call `replicate()` on important data.
Add a boolean flag to the Compute message, stating that the task was previously in memory at some point and it's now being recomputed.
When the task ends successfully on the worker, instead of logging its granular metrics we will log a lump sum under the `("execute", , "recompute", "seconds")` label. This is an equivalent treatment to when a task fails and we log a lump sum under the `("execute", , "failed", "seconds")` label, which was introduced in #7586.
Contributor guide
Assessment
This issue has not been assessed yet.