Add back worker reconnection
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
If the network connection between the worker and scheduler was broken, workers used to try to re-connect and negotiate their state with the scheduler.
It turned out that the logic around re-estabilshing the network connection (https://github.com/dask/distributed/issues/5481), re-negotiating the state (https://github.com/dask/distributed/pull/6341), and handling the disconnect on the scheduler side (https://github.com/dask/distributed/issues/6354) was all buggy and a source of deadlocks. Though disruptive, for short-term stability, we opted to remove the reconnection option entirely (https://github.com/dask/distributed/issues/6350).
However, in the long term, we do want workers to be resilient to temporary network failures. We'll want to add worker reconnection back in once contracts around `BatchedSend` and worker disconnection are tightened up.
Requires:
* https://github.com/dask/distributed/issues/6389
* https://github.com/dask/distributed/issues/6390
* and probably some other things
Note that I'm intentionally not tracking this in https://github.com/dask/distributed/issues/6384, since those are only meant to be short-term tasks. This is likely not something we'll tackle for a bit.
Contributor guide
Assessment
This issue has not been assessed yet.