dask / dask/dask-gateway

Gateway fails to shutdown if kernel restarted while the graph is executing

Open
#155 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
148
Forks
93
PR merge metrics
No merged PRs in 30d

Description

I've encountered this situation a few times now where I realize right after I start a computation that my worker size is too small and I know the task graph will fail. So I instinctually restart my jupyter kernel, up the worker size and create a new cluster (on GKE with dask-gateway). If I restart the kernel while the graph is executing, the cluster that I had created is not auto-killed, I suspect because the client can't signal to the gateway that it should kill the cluster. I think it's a fair assumption that users will restart their kernel during a computation, so we likely need a pattern for how to best handle this. Any thoughts on how this condition should be handled?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.