dask / dask/distributed

Fine performance metrics: Meter task re-execution after losing a worker

Open
#7,676 0 comments 0 reactions 0 assignees View on GitHub
diagnostics
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

- Part of #7665
- Complements #7678

When a worker dies and you lose tasks in `memory` state, they transition back to `released` on the scheduler and are re-computed somewhere else.

We would like to know how much time we spent re-computing tasks after a worker dies. This could inform the user e.g. to call `replicate()` on important data.

Add a boolean flag to the Compute message, stating that the task was previously in memory at some point and it's now being recomputed.
When the task ends successfully on the worker, instead of logging its granular metrics we will log a lump sum under the `("execute", , "recompute", "seconds")` label. This is an equivalent treatment to when a task fails and we log a lump sum under the `("execute", , "failed", "seconds")` label, which was introduced in #7586.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.