dask / dask/distributed

Improve handling of tasks failing/succeeding on unexpected workers

Open
#6,956 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

This issue is a follow-up on an offline discussion centering around https://github.com/dask/distributed/pull/6884#issuecomment-1224775494 and #6939 with @crusaderky and @gjoseph92 where we encountered some issues with the way we currently handle tasks failing/succeeding on unexpected workers.

In `transition_processing_memory`, when encountering an unexpected worker that has successfully completed a task, we send a `cancel-compute` message to the worker that was supposed to process this task assuming that the unexpected worker would hold onto our result for us.

https://github.com/dask/distributed/blob/c15a10e87ca5d03e62f0ad4f38adb63163522979/distributed/scheduler.py#L1987-L2001

However, this series of events _should_ only be triggered in a chain of events where `free-keys` message for that task should already be on its way to the unexpected worker, causing it to remove the result. This leaves us with no worker holding on to the task/result.

While we do suspect this chain of events to unfold, we need a test verifying it. If this is indeed the case, we should not send the `cancel-compute` message to `ts.processing_on`, but rather make sure that we clean up anything related to the work happening on the unexpected worker.

In `stimulus_task_erred`, we don't check whether the task `erred` on an unexpected worker at all and run through the retry-logic. In https://github.com/dask/distributed/pull/6884#issuecomment-1217824082, @fjetter suggested instead ignoring that the task `erred` on an unexpected worker and continuing on as if nothing happened. This general approach is reasonable as the unexpected worker suggests something faulty going on. We might need to do additional work to ensure that the unexpected worker is reset, e.g., sending a `free-keys` message to the unexpected workers and we want to log a warning. To ensure everything is reset correctly, we need a test that verifies the intended behavior.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.