[BUG] Comms not robust to worker failures
Nobody has claimed this yet.
- Dominant language
- Cuda
- Stars
- 1k
- Forks
- 251
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 31
Description
I'm unsure whether this is really a problem with the comms or whether this information needs to be better provided within Dask, however if a worker disconnects, the scheduler seems to still return the worker as a candidate for scheduling tasks and this can cause the comms initialization to hang while the client tries to run a task directly on that worker (since client.run() is used)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing a worker disconnect during communications initialization, then trace how the scheduler still returns that worker as a candidate and how client.run() is used. Done means initialization no longer hangs when a candidate worker has disconnected.
Written by the indexing model from the issue text.
Assessment
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100