dask / dask/distributed

Confusing Nanny on_exit callback structure

Open
#7,321 0 comments 0 reactions 0 assignees View on GitHub
asyncio hygiene
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

The Nanny is managing a subprocess in which a Worker is started. If that process exits a cascade of on_exit callbacks are triggered.

The order in which things happen is

- `AsyncProcess._on_exit`
This one is not doing a lot. It sets an event s.t. the process is joinable. Then it triggers another on_exit
- `WorkerProcess._on_exit` is just calling `WorkerProcess.mark_stopped`
- `WorkerProcess.mark_stopped` is resetting some state in `WorkerProcess` and is calling another on_exit
- `Nanny._on_worker_exit_sync` is scheduling a coroutine on the loop which is the next on_exit
- `Nanny._on_worker_exit` is unregistering the worker from the scheduler and if need be restarts the worker process

This chain of events is not only confusing but also subject to race conditions. Particularly that the final, most relevant on_exit callback is scheduled with a `loop.call_soon` allows for various race conditions.

These race conditions are currently not a direct issue. Most race conditions are actually buffered by various idempotent implementations of close/start but once we touch this structure, this is getting a bit shaky.

I debugged this during the investigation of https://github.com/dask/distributed/issues/7312 but this chain is not directly causing the issue. This issue is mostly to document the situation.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.