SpecClusters may associate the wrong name for a worker
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
I'm currently trying to track down an issue where adaptive scaling sometimes appears to shut down the wrong workers. I'm not 100% sure that this is the cause of what I'm observing, but this code does seem a bit suspicious, so I wanted to open up an issue for discussion.
The `SpecCluster` keeps 2 dictionaries, one mapping names to `worker_spec` objects, and another mapping names to `worker` objects. During adaptive scaling, the `SpecCluster` modifies the `worker_spec`, and `_correct_state_interval` will perform actions to reconcile `workers` with `worker_spec`. Either by tearing down or constructing new workers.
`_correct_state_interval` loops over new workers from `worker_spec`, and instantiates them.
It does this, by iterating through a set: [to_open](https://github.com/dask/distributed/blob/1be9265ac11876df766bb8bd6d6eb519d04d3bac/distributed/deploy/spec.py#L348). For each name in to_open, it creates a worker, and appends that worker to a list. Finally it `zips` `to_open`, and `workers` together.
I don't believe iteration order of sets is guaranteed? Which would mean that it's possible for the keys in the `workers` dictionary, which should be the names of the workers, to get out of sync with the actual names of the workers.
https://github.com/dask/distributed/blob/1be9265ac11876df766bb8bd6d6eb519d04d3bac/distributed/deploy/spec.py#L366
I believe the simplest fix would just be to convert `to_open` to a list.
Contributor guide
Assessment
This issue has not been assessed yet.