dask / dask/distributed

Missing input data for a task should fail worker, not the task

Open
#6,142 0 comments 0 reactions 0 assignees View on GitHub
stability
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

If an input to a task is not actually in `self.data`, this currently manifests as the task failing (as though it's a user error). This in fact indicates a serious worker state issue, and should probably cause the entire worker to shut down.

See https://github.com/dask/distributed/issues/6125#issuecomment-1099877076 for a traceback of how an invalid transition led to a task executing where its inputs weren't all in memory yet. The `df.compute()` in client code failed as though this was a normal error, with the message `KeyError: "('split-shuffle-1-b4961b03aa9e8bec7c581d2dc337f717', 10, (3, 9))"`.

The problem is this overly-generous try/except in `Worker.execute`, which treats any problem in the `try` block as an issue with the task, including `_prepare_args_for_execution`, which looks up the input keys: https://github.com/dask/distributed/blob/68319f377da81057de8cfd4277f2448d796c592a/distributed/worker.py#L3488-L3601

Most problems that could happen there would be user error, but not all of them. We should be more selective?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.