SpecCluster calls scheduler.retire_workers incorrectly
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
As part of `_correct_state_internal()`, the SpecCluster calls `retire_workers` to gracefully shut down any `to_close` workers before they are closed.
https://github.com/dask/distributed/blob/7a3ea4c34ae0af6af1137fe3e3bda927aa9eac26/distributed/deploy/spec.py#L330-L332
There are two bugs in this section of code.
1) `to_close` contains the _names_ of the workers, not the addresses. See the [fn definition for `retire_workers`](https://github.com/dask/distributed/blob/7a3ea4c34ae0af6af1137fe3e3bda927aa9eac26/distributed/scheduler.py#L6559), you'll see that the call should be `retire_workers(names=list(to_close))`, because the `workers` argument is supposed to take addresses.
2) The SpecCluster in some aspects has a concept of worker grouping. That is, a single worker at the cluster level can spawn more than one worker process at the scheduler level. When calling on the scheduler to retire the given workers, the SpecCluster should check its specs to determine if the given worker is a group worker, and populate a list of suffixed named to send to the retire_workers function.
**Environment**:
- Distributed version: 21.11.2
- Python version: 3.9
- Operating System: Ubuntu 20.04
- Install method: pip
Contributor guide
Assessment
This issue has not been assessed yet.