dask / dask/distributed

Tasks with worker restrictions get stuck in `no-worker` when required worker is removed

Open
#7,346 9 comments 0 reactions 0 assignees View on GitHub
discussion
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**Problem**

When tasks are restricted to run on a specific worker using `worker_restrictions`, they transition to `no-worker` if the specified worker is removed. The task will remain in `no-worker` until a worker with the _same_ address rejoins. I see two use cases for this behavior:

1. Restricting a task to run on a specific machine (with specific hardware)
* In this case, the user may be better served by using `host_restrictions` or `resource_restrictions`
2. Restricting a task to run on a specific worker instance
* For example, we restrict tasks to run on specific worker instances in the P2PShuffle implementation
* In this case, we want tasks to fail as soon as the instance is removed instead of remaining in `no-worker` indefinitely

Am I missing a use case here?

Depending on your deployment system, another worker with the same address _may_ join the cluster, which would resolve the deadlock of use case #1, but create trouble for tasks that want to achieve use case #2 as another manifestation of #6392.

**Possible solutions**

1. Adjust `worker_restrictions` to use IDs instead of addresses to ensure uniqueness and transition tasks to `erred` instead of `no-worker`. This would be a breaking change and I do not know if anybody relies on the current behavior.
2. Introduce another type of restrictions based on IDs that restricts workers to currently existing worker instances via IDs and implements the behavior outlined above.

cc @fjetter

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.