Tasks with worker restrictions get stuck in `no-worker` when required worker is removed
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
**Problem**
When tasks are restricted to run on a specific worker using `worker_restrictions`, they transition to `no-worker` if the specified worker is removed. The task will remain in `no-worker` until a worker with the _same_ address rejoins. I see two use cases for this behavior:
1. Restricting a task to run on a specific machine (with specific hardware)
* In this case, the user may be better served by using `host_restrictions` or `resource_restrictions`
2. Restricting a task to run on a specific worker instance
* For example, we restrict tasks to run on specific worker instances in the P2PShuffle implementation
* In this case, we want tasks to fail as soon as the instance is removed instead of remaining in `no-worker` indefinitely
Am I missing a use case here?
Depending on your deployment system, another worker with the same address _may_ join the cluster, which would resolve the deadlock of use case #1, but create trouble for tasks that want to achieve use case #2 as another manifestation of #6392.
**Possible solutions**
1. Adjust `worker_restrictions` to use IDs instead of addresses to ensure uniqueness and transition tasks to `erred` instead of `no-worker`. This would be a breaking change and I do not know if anybody relies on the current behavior.
2. Introduce another type of restrictions based on IDs that restricts workers to currently existing worker instances via IDs and implements the behavior outlined above.
cc @fjetter
Contributor guide
Assessment
This issue has not been assessed yet.