dask / dask/distributed

Restart worker via Nanny on connection failure

Open
#6,387 1 comment 0 reactions 0 assignees View on GitHub
enhancement hygiene
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

With https://github.com/dask/distributed/pull/6361, any temporary network disconnect will shut down the worker.

If we have a Nanny around, we could at least let the Nanny restart the worker, so it can try to reconnect from a fresh state. Currently, we tell the Nanny to close gracefully.

There are a couple of things to think about:
* Make sure that, if the scheduler has asked us to close and then closes the connection, the Nanny doesn't try to restart
* The Nanny also [reports worker loss to the scheduler](https://github.com/dask/distributed/blob/33fc50ca9817216bb4105b68f5e0859ebfb80fdb/distributed/nanny.py#L287-L294). This _shouldn't_ happen until the worker process is already closed, which means the network connection _should_ already be closed, so the scheduler _should_ have already [removed the worker itself](https://github.com/gjoseph92/distributed/blob/305a23fa4261d7b8193daeec6ae6012f0af956c3/distributed/scheduler.py#L4792-L4797), but think about race conditions here.
* https://github.com/dask/distributed/issues/6386

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.