dask / dask/distributed

Resubmit tasks on Executor reconnect

Open
#436 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Currently when a scheduler goes down and comes back empty all futures on all executors are cancelled/lost.

However, the Executor could choose to hold on to how it produced all futures. This would allow it to resubmit all computations. There are pros and cons to this:
- Pro: This would establish the weakest form of full resilience to scheduler failure.
- Con: There are two copies of the dask graph lying around. Suddenly the Executor gets more complex.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.