dask / dask/distributed

Add back worker reconnection

Open
#6,391 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

If the network connection between the worker and scheduler was broken, workers used to try to re-connect and negotiate their state with the scheduler.

It turned out that the logic around re-estabilshing the network connection (https://github.com/dask/distributed/issues/5481), re-negotiating the state (https://github.com/dask/distributed/pull/6341), and handling the disconnect on the scheduler side (https://github.com/dask/distributed/issues/6354) was all buggy and a source of deadlocks. Though disruptive, for short-term stability, we opted to remove the reconnection option entirely (https://github.com/dask/distributed/issues/6350).

However, in the long term, we do want workers to be resilient to temporary network failures. We'll want to add worker reconnection back in once contracts around `BatchedSend` and worker disconnection are tightened up.

Requires:
* https://github.com/dask/distributed/issues/6389
* https://github.com/dask/distributed/issues/6390
* and probably some other things

Note that I'm intentionally not tracking this in https://github.com/dask/distributed/issues/6384, since those are only meant to be short-term tasks. This is likely not something we'll tackle for a bit.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.