dask / dask/distributed

Race condition in `SpecCluster.close` when closing while upscaling

Open
#5,876 0 comments 0 reactions 0 assignees View on GitHub
adaptive
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

There is a race condition in `SpecCluster.close` that can lock potentially indefinitely if a cluster is closing while instances are spawned at the same time. It is not clear, yet, if this deadlock resolves itself given enough time.

This has been diagnosed as a root cause for some of the flaky tests, see https://github.com/dask/distributed/issues/4859#issuecomment-854705100

As a user I would expect an ongoing scale up attempt to be canceled during cluster closing and the closing to take care of cleaning up all already created instances.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.