dask / dask/distributed

Allow NannyPlugins to handle worker process terminations

Open
#5,465 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Sometimes task code can cause workers to segfault. It's unfortunate, and it causes a lot of cluster instability. These segfaults often surface as strange comms related exceptions on the scheduler. Unfortunately it's really hard to currently detect segfaults and inform the user that something terrible has happened and they should fix their code.

One possible way would be to add a hook to NannyPlugins that is called in the event of a worker shutdown with the exit status code. Then the nanny could take some action (log a message, send an event, trigger something on the scheduler) which can be used to more reliably do _something_ with the information.

In our specific use case, we'd like to call `faulthandler.enable()` from a WorkerPlugin and give it a specific unique path. If a worker dies in a non-expected way, we'd like the Nanny to upload this faulthandler output file to object storage for later diagnosis and increment a counter on the scheduler which can be used to display useful information to the user on cluster termination.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.