dask / dask/distributed

"can't pickle thread.lock objects" when working with published dataframe

Open
#1,556 11 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

We're using dask distributed scheduler with multiprocessing workers on an EC2 cluster.
dask 0.15.4 and distributed 1.19.3

I'm trying to publish named dataset (dataframe) and then retrieve and continue working on it. Basically:
```
frame = df.read_csv(url, ...)
client.publish_dataset(ds_name=frame)
ds = client.get_dataset(ds_name)
client.compute(ds)
```

This results in `'TypeError: can't pickle thread.lock objects'` error.

I suppose this might be related to:
https://github.com/dask/distributed/issues/780
https://github.com/dask/dask/issues/1683
https://github.com/dask/distributed/issues/539

I don't know how to work around this issue because read_csv() doesn't seem to accept lock argument.

full traceback:
[traceback.txt](https://github.com/dask/distributed/files/1470670/traceback.txt)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.