planned worker death / self-scale down
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
It would be nice if workers *themselves* could signal that they are about to shutdown and have the scheduler do the preparatory scaling down i.e. move completed results to other workers. I'm imagining e.g. a `--scale-down-on-exit` command line option.
#### Motivating scenario
I currently have a local scheduler + jupyter session etc with very long running computations. I submit 100s+ dask workers to a batch system which form a dynamic pool. This is largely a **great** setup (thank you distributed), which allows saturation of the available resources. But since it is a batch system favoring time limits less than the overall computation time, the workers are regularly killed, and generally the final workers are all different to the initial ones.
The issue is that killed workers take their results with them, which then have to be recomputed. So in the situation above, the computation can never complete unless the results are eagerly gathered. If the workers could mark themselves to be scaled down, e.g. after receiving SIGTERM just before their time limit, then results could propagate.
#
If there is already a simple way to do this that I have missed - apologies and please close. `replicate` seems like it might do the job, but I couldn't get it to work as needed, and the the warning about not replicating to future workers seemed relevant. Currently I just gather everything to the local process which is fine as the computations I'm performing are very computationally intensive but produce small results.
Contributor guide
Research direction
No file or test is named. Start by tracing the worker SIGTERM/shutdown path and the scheduler's handling of departing workers, then compare the requested behavior with replicate. Done means a worker can signal impending exit, completed results move to other workers, and the behavior is available through a --scale-down-on-exit option.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100