dask / dask/distributed

P2P does not play nicely with loose restrictions

Open
#8,407 0 comments 0 reactions 0 assignees View on GitHub
bug shuffle
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

When P2P has "loose" restrictions, the scheduler will not enforce the worker restrictions P2P relies on for `unpack` tasks. As a result, P2P will be caught in a rescheduling loop until every task eventually ends up on the correct worker by chance.

Implementation notes:
We can't simply unset loose restrictions. If there are unsatisfiable restrictions, P2P will be stuck forever.
If we have loose worker restrictions, P2P should not limit itself to these workers but use the entire pool of workers.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.