dask / dask/distributed

P2P tasks log compute failures even if they are later restarted

Open
#8,679 0 comments 0 reactions 0 assignees View on GitHub
enhancement shuffle
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

# Problem

There is a race condition in P2P that causes tasks to log compute failures on the worker even though those tasks will get restarted later on and then succeed. This happens when:

1. A worker involved in the P2P operation is removed
2. We restart the P2P operation on the scheduler and schedule the messages to be sent to the workers
3. A task on worker A is not cancelled yet, but its RPC calls fail because the remote worker B has already closed the shuffle run, throwing a `P2PConsistencyError`
4. The task raises the `P2PConsistencyError` and fails while still seen as `executing` by worker A, which causes the error to get logged.

# Solution

Instead of failing directly on a `P2PConsistencyError`, the task could double-check with the scheduler whether its shuffle run is still supposed to be active. If not, it could instead silently succeed as the result will get rejected by the scheduler as outdated.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.