Restart worker via Nanny on connection failure
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
With https://github.com/dask/distributed/pull/6361, any temporary network disconnect will shut down the worker.
If we have a Nanny around, we could at least let the Nanny restart the worker, so it can try to reconnect from a fresh state. Currently, we tell the Nanny to close gracefully.
There are a couple of things to think about:
* Make sure that, if the scheduler has asked us to close and then closes the connection, the Nanny doesn't try to restart
* The Nanny also [reports worker loss to the scheduler](https://github.com/dask/distributed/blob/33fc50ca9817216bb4105b68f5e0859ebfb80fdb/distributed/nanny.py#L287-L294). This _shouldn't_ happen until the worker process is already closed, which means the network connection _should_ already be closed, so the scheduler _should_ have already [removed the worker itself](https://github.com/gjoseph92/distributed/blob/305a23fa4261d7b8193daeec6ae6012f0af956c3/distributed/scheduler.py#L4792-L4797), but think about race conditions here.
* https://github.com/dask/distributed/issues/6386
Contributor guide
Assessment
This issue has not been assessed yet.