Allow NannyPlugins to handle worker process terminations
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Sometimes task code can cause workers to segfault. It's unfortunate, and it causes a lot of cluster instability. These segfaults often surface as strange comms related exceptions on the scheduler. Unfortunately it's really hard to currently detect segfaults and inform the user that something terrible has happened and they should fix their code.
One possible way would be to add a hook to NannyPlugins that is called in the event of a worker shutdown with the exit status code. Then the nanny could take some action (log a message, send an event, trigger something on the scheduler) which can be used to more reliably do _something_ with the information.
In our specific use case, we'd like to call `faulthandler.enable()` from a WorkerPlugin and give it a specific unique path. If a worker dies in a non-expected way, we'd like the Nanny to upload this faulthandler output file to object storage for later diagnosis and increment a counter on the scheduler which can be used to display useful information to the user on cluster termination.
Contributor guide
Assessment
This issue has not been assessed yet.