dask / dask/distributed

Recommendation Requested / Observed mismatch

Open
#8,119 2 comments 0 reactions 0 assignees View on GitHub
adaptive bug
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

When the adaptive core gets a target value that recommends a scale-down, it always appears to take the first worker, as defined here: https://github.com/dask/distributed/blob/716d5260ce8b9f8c5e46fbdfba54f0805f40fd9a/distributed/deploy/adaptive_core.py#L192

This is because `requested` and `observed` contain completely different data.
`requested` takes the name of the worker, which is indicated by the clusters as an incremental integer.
https://github.com/dask/distributed/blob/716d5260ce8b9f8c5e46fbdfba54f0805f40fd9a/distributed/deploy/spec.py#L550

However, the `observed` names, or the names the scheduler gets from the workers, appear to be the addresses of the workers.
This results in a mismatch as the sets are compared, and as there is no overlap, the adaptive core assumes that it is still awaiting some workers, and thus can kill the not-yet-arrived workers. This is counterproductive, as this causes the adaptive algorithm to kill based on ordering, rather than idle behaviour.
![Screenshot 2023-08-21 at 10 25 37](https://github.com/dask/distributed/assets/7126259/58e86bff-699e-4661-a352-1048699a4e6f)

**Environment**:

- Dask version: 2023.3.0
- Python version: 3.11
- Operating System: Ubuntu 22.04 (docker)
- Install method (conda, pip, source): pip

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.