cloudpipe / cloudpipe/cloudpickle

Failure to pickle with type annotations (version 1.2.2, with repro)

Open
#312 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.9k
Forks
195
Avg merge
1d 10h
Merged PRs (30d)
1

Description

Even with #299, there seems to be a corner case being hit in the below repro on version 1.2.2 that gives rise to the following error:

`
PicklingError: Can't pickle typing.Union[pandas.core.frame.DataFrame, pandas.core.series.Series, numpy.ndarray, str, NoneType]: it's not the same object as typing.Union
`

Repro is following:

```
from typing import Optional, Union

import pandas as pd
import numpy as np
import inspect

import cloudpickle

class AddStuff:
def __init__(self, other: Optional[Union[pd.DataFrame, pd.Series, np.ndarray, str]]=None):
self._other = other

def add(self, data):
return self._other + data

def class_to_func(klass):
sig = inspect.signature(klass.__init__)

def wrapped(data, *args, **kwargs):
t = klass(*args, **kwargs)
return t.add(data)

data_param = inspect.Parameter(
'data', inspect.Parameter.POSITIONAL_OR_KEYWORD)

new_parameters = [data_param] + list(sig.parameters.values())[1:]
# ***** uncomment next line out, for things to work *****
# new_parameters = _remove_annotations(new_parameters)

new_sig = sig.replace(parameters=tuple(new_parameters))

wrapped.__signature__ = new_sig

return wrapped

def _remove_annotations(parameters):
return list(map(lambda p: p.replace(annotation=inspect.Parameter.empty),
parameters))


df = pd.DataFrame(np.random.rand(2,2))
adder = class_to_func(AddStuff)

v2 = AddStuff(df).add(df)
v1 = adder(df, df)

# prints True
print(v1.equals(v2))

# ok
cloudpickle.dumps(AddStuff)

# blows up! unless we use _remove_annotations
cloudpickle.dumps(adder)
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.