Fine performance metrics: Meter wasted partial compute time after losing a worker
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
- Part of #7665
- Blocked by #7666
- Blocked by #7677
- Complements #7676
When we lose a worker, any time spent partially executing a task is lost. The task transitions back from `processing` to `released` on the scheduler, so that it may be executed somewhere else.
When the scheduler receives metrics from the heartbeat (#7666), it normally immediately forgets which worker they come from. It should make an execption for `currently-running` (#7677) and keep track of which worker they come from.
When a worker dies, it should subtract all `currently-running` time for the worker and reclassify it as `("execute", , "killed-worker", "seconds")`.
Additionally, it should add to this measure all tasks that were currently executing as of the last received heartbeat, minus those that completed in the meantime, multiplied by the time between the last heartbeat and the worker death.
Contributor guide
Assessment
This issue has not been assessed yet.