dask / dask/distributed

Unable to serialize function with closure including object with custom metadata

Open
#4,092 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Example:

```python
import pydantic
from distributed.worker import dumps_function

class Bar(pydantic.BaseModel):
pass

def f(bar: Bar):
def inner():
print(bar)

dumps_function(inner)

if __name__ == '__main__':
f(Bar())
```

This works on python 3.8, but not 3.6. It's caused by the upgrade of cloudpickle from 1.3.0 to 1.5.0 in a8a7586507fb73a08d5a80a1bd79a35ec4af3794

You can't replace the serializer used by `dumps_function`, so I can't replace it with dill. Dill works.

**What happened**:

Exception:
```
_pickle.PicklingError: Cloudpickle Error: Unknown type
```

**What you expected to happen**:

No exception, pickled function.

**Minimal Complete Verifiable Example**:

See above

**Anything else we need to know?**:

I've tried to report it against `cloudpickle` https://github.com/cloudpipe/cloudpickle/issues/393

**Environment**:

- Dask version: fails: `dask[complete]==2.25.0`, `distributed==2.25.0`, works in `distributed==2.16.0`
- Python version: works in 3.8, fails in 3.6
- Operating System: Windows
- Install method (conda, pip, source): pip

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.