"can't pickle thread.lock objects" when working with published dataframe
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
We're using dask distributed scheduler with multiprocessing workers on an EC2 cluster.
dask 0.15.4 and distributed 1.19.3
I'm trying to publish named dataset (dataframe) and then retrieve and continue working on it. Basically:
```
frame = df.read_csv(url, ...)
client.publish_dataset(ds_name=frame)
ds = client.get_dataset(ds_name)
client.compute(ds)
```
This results in `'TypeError: can't pickle thread.lock objects'` error.
I suppose this might be related to:
https://github.com/dask/distributed/issues/780
https://github.com/dask/dask/issues/1683
https://github.com/dask/distributed/issues/539
I don't know how to work around this issue because read_csv() doesn't seem to accept lock argument.
full traceback:
[traceback.txt](https://github.com/dask/distributed/files/1470670/traceback.txt)
Contributor guide
Assessment
This issue has not been assessed yet.