dask / dask/distributed

Scattered data pickles on the client but fails to pickle on evict()

Open
#5,970 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

# Use case
A client invokes scatter() on one or more keys.
All of them successfully pickle on the client and are sent over to the worker, where they successfully unpickle.
One of the values is individually larger than the memory_limit * target, so it is immediately pickled again and spilled.
However, for whatever reason it fails to pickle a second time - e.g. because the pickling algorithm is unstable, or because it changes behaviour depending on the interpreter state.

# Acceptable behaviours
1. The whole scatter operation fails, and all keys are forgotten. On the client side, scatter() re-raises the pickle exception.
2. scatter succeeds. The key that failed to pickle is immediately marked as "error" on the client side. Trying to retrieve it re-raises the pickle exception. (this behaviour would be coherent with submitted tasks that fail to pickle on evict()).

# Actual behaviour
scatter succeeds.
The failing key is marked as "finished" on the client side, but it is not in Worker.data.
Trying to retrieve it raises a non-descript CancelledError on the client.

POC and failing unit test: #5969

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.