dask / dask/distributed

SpecCluster calls scheduler.retire_workers incorrectly

Open
#5,577 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

As part of `_correct_state_internal()`, the SpecCluster calls `retire_workers` to gracefully shut down any `to_close` workers before they are closed.

https://github.com/dask/distributed/blob/7a3ea4c34ae0af6af1137fe3e3bda927aa9eac26/distributed/deploy/spec.py#L330-L332

There are two bugs in this section of code.
1) `to_close` contains the _names_ of the workers, not the addresses. See the [fn definition for `retire_workers`](https://github.com/dask/distributed/blob/7a3ea4c34ae0af6af1137fe3e3bda927aa9eac26/distributed/scheduler.py#L6559), you'll see that the call should be `retire_workers(names=list(to_close))`, because the `workers` argument is supposed to take addresses.

2) The SpecCluster in some aspects has a concept of worker grouping. That is, a single worker at the cluster level can spawn more than one worker process at the scheduler level. When calling on the scheduler to retire the given workers, the SpecCluster should check its specs to determine if the given worker is a group worker, and populate a list of suffixed named to send to the retire_workers function.

**Environment**:

- Distributed version: 21.11.2
- Python version: 3.9
- Operating System: Ubuntu 20.04
- Install method: pip

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.