Missing input data for a task should fail worker, not the task
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
If an input to a task is not actually in `self.data`, this currently manifests as the task failing (as though it's a user error). This in fact indicates a serious worker state issue, and should probably cause the entire worker to shut down.
See https://github.com/dask/distributed/issues/6125#issuecomment-1099877076 for a traceback of how an invalid transition led to a task executing where its inputs weren't all in memory yet. The `df.compute()` in client code failed as though this was a normal error, with the message `KeyError: "('split-shuffle-1-b4961b03aa9e8bec7c581d2dc337f717', 10, (3, 9))"`.
The problem is this overly-generous try/except in `Worker.execute`, which treats any problem in the `try` block as an issue with the task, including `_prepare_args_for_execution`, which looks up the input keys: https://github.com/dask/distributed/blob/68319f377da81057de8cfd4277f2448d796c592a/distributed/worker.py#L3488-L3601
Most problems that could happen there would be user error, but not all of them. We should be more selective?
Contributor guide
Assessment
This issue has not been assessed yet.