Add option to shutdown cluster if client disconnects after a timeout
- Dominant language
- Python
- Stars
- 148
- Forks
- 93
- PR merge metrics
- No merged PRs in 30d
Description
When a `GatewayCluster` object is cleaned up, by default it will shutdown the associated cluster. In a perfect scenario this works fine, but because this requires an external API call and is implemented using `__del__` (and `atexit` as a fallback), it comes with a few caveats.
- It can't work if the interpreter is killed (no way to register a signal handler for SIGKILL). It alsp currently won't work with SIGTERM, but we could register a signal handler for that. I'm a bit hesitant to (since that's a global change to the interpreter), but it could be done.
One case where this comes up periodically is if the user restarts their notebook kernel *while it's computing something*. In this case the kernel is terminated in a non-graceful manner and the atexit handlers don't run (see https://github.com/dask/dask-gateway/issues/155).
- It can't work in the presence of segfaults.
- It can't work in the presence of network failures
In common cases though (normal Python shutdown is allowed to succeed), the existing mechanism works. Other dask cluster managers that support external schedulers (dask-yarn, dask-kubernetes) will have the same problems, this isn't specific to dask-gateway.
---
I expect the failure mode here to be uncommon, but still occur often enough that we'll want a way to handle it. We currently support shutting down idle clusters after a timeout (with [`ClusterConfig.idle_timeout`](https://gateway.dask.org/api-server.html#c.ClusterConfig.idle_timeout)), but this also affects clusters with active connections (i.e. if you sit and think for longer than this timeout, your cluster will shutdown).
A different option might be to add another timeout for disconnect from any external clients. This could be set to a smaller value (perhaps 30-120s seems reasonable) to catch disconnects earlier. This would work in all cases when the client session closed.
There's some logic decisions to make here:
- Do we shutdown only if no external clients *and* the cluster is idle? Or even if the cluster is still computing?
- Currently determining which clients are external clients (meaning not a `worker_client`) isn't possible - we don't have access to that information. This would require an upstream change to distributed.
- Alternatively we could shutdown if the connection is dropped (after a timeout) from the `GatewayCluster` object. This could be handled without upstream changes, but I'm not sure if this is the right shutdown condition.
Contributor guide
Assessment
This issue has not been assessed yet.