dask / dask/distributed

SpecClusters may associate the wrong name for a worker

Open
#5,135 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I'm currently trying to track down an issue where adaptive scaling sometimes appears to shut down the wrong workers. I'm not 100% sure that this is the cause of what I'm observing, but this code does seem a bit suspicious, so I wanted to open up an issue for discussion.

The `SpecCluster` keeps 2 dictionaries, one mapping names to `worker_spec` objects, and another mapping names to `worker` objects. During adaptive scaling, the `SpecCluster` modifies the `worker_spec`, and `_correct_state_interval` will perform actions to reconcile `workers` with `worker_spec`. Either by tearing down or constructing new workers.

`_correct_state_interval` loops over new workers from `worker_spec`, and instantiates them.

It does this, by iterating through a set: [to_open](https://github.com/dask/distributed/blob/1be9265ac11876df766bb8bd6d6eb519d04d3bac/distributed/deploy/spec.py#L348). For each name in to_open, it creates a worker, and appends that worker to a list. Finally it `zips` `to_open`, and `workers` together.

I don't believe iteration order of sets is guaranteed? Which would mean that it's possible for the keys in the `workers` dictionary, which should be the names of the workers, to get out of sync with the actual names of the workers.

https://github.com/dask/distributed/blob/1be9265ac11876df766bb8bd6d6eb519d04d3bac/distributed/deploy/spec.py#L366

I believe the simplest fix would just be to convert `to_open` to a list.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.