dask / dask/distributed

Expose workers_to_close as a HTTP route

Open
#6,416 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

With #6270 we now have a first set of HTTP routes that are exposed on the scheduler. Currently there are two modes for the `/api/v1/retire_workers` route:

1. Retire n workers based on the workers_to_close method of the scheduler
2. Retire the provided workers based on their address

But there is no API yet that exposes the `workers_to_close` method directly.
At least in our use case that would be really helpful though since our graceful shutdown implementation has two phases:

1. Retrieve workers to close from the scheduler and do some custom graceful shutdown logic
2. Actually make the retire_worker call to the scheduler.

### Background

We have implemented some custom shutdown logic for Dask workers on Kubernetes. When we need to scale down the number of Dask workers we query the scheduler for which workers we should shutdown. These workers are then marked for termination (via an annotation on the deployment object of the worker) and then retired at the scheduler. After a configured time the worker deployments are then deleted.

When we retire the workers at the scheduler we set the `close_workers` option to `False` otherwise the workers would be restarted by Kubernetes and try to reconnect to the scheduler. But this also means we have to have a way to clean up all the retired workers, hence the annotation on the deployments. We also don't want to set the annotations after we made the retire worker call to avoid leaving behind any zombie workers that are not useable anymore.

@jacobtomlinson

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.