cloudpipe / cloudpipe/cloudpickle

Cloudpickle cannot pickle cythonized methods

Open
#259 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.9k
Forks
195
Avg merge
1d 10h
Merged PRs (30d)
1

Description

I'm using Dask to parallelize computing. And my code is compiled with cython.

When computing, Dask actually asks cloudpickle to dump the compute method, but this operation fails.

The problem is that it looks like cloudpickle cannot dump methods of a class if this class has been compiled with cython. But such methods can be dumped using pickle.

**How to reproduce:**

Here is the code of the class below. I'd like to pickle the method `my_method` from an object of this class:

```python
class MyClass:

def __init__(self, a):
self.a = a

def my_method(self, b):
return self.a + b
```

The following code works properly when importing from the python module:

```python
import cloudpickle
from module import MyClass

my_obj = MyClass(a=10)
cloudpickle.dumps(my_obj.my_method)
```

But this fails if the module is cythonized.

If the module is cythonized and `MyClass` is imported from there, call to `cloudpickle.dumps(my_obj.my_method)` generates the following error:

```
>>> cloudpickle.dumps(my_obj.my_method)
Traceback (most recent call last):
File "", line 1, in
File "/home/marc.beillevaire/conda/envs/test_cythn/lib/python3.7/site-packages/cloudpickle/cloudpickle.py", line 961, in dumps
cp.dump(obj)
File "/home/marc.beillevaire/conda/envs/test_cythn/lib/python3.7/site-packages/cloudpickle/cloudpickle.py", line 267, in dump
return Pickler.dump(self, obj)
File "/home/marc.beillevaire/conda/envs/test_cythn/lib/python3.7/pickle.py", line 437, in dump
self.save(obj)
File "/home/marc.beillevaire/conda/envs/test_cythn/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/home/marc.beillevaire/conda/envs/test_cythn/lib/python3.7/site-packages/cloudpickle/cloudpickle.py", line 727, in save_instancemethod
self.save_reduce(types.MethodType, (obj.__func__, obj.__self__), obj=obj)
File "/home/marc.beillevaire/conda/envs/test_cythn/lib/python3.7/pickle.py", line 638, in save_reduce
save(args)
File "/home/marc.beillevaire/conda/envs/test_cythn/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/home/marc.beillevaire/conda/envs/test_cythn/lib/python3.7/pickle.py", line 771, in save_tuple
save(element)
File "/home/marc.beillevaire/conda/envs/test_cythn/lib/python3.7/pickle.py", line 535, in save
self.save_global(obj, rv)
File "/home/marc.beillevaire/conda/envs/test_cythn/lib/python3.7/site-packages/cloudpickle/cloudpickle.py", line 705, in save_global
return Pickler.save_global(self, obj, name=name)
File "/home/marc.beillevaire/conda/envs/test_cythn/lib/python3.7/pickle.py", line 957, in save_global
(obj, module_name, name)) from None
_pickle.PicklingError: Can't pickle : it's not found as module.my_method
```

Here is the simple setup.py file I'm using to generate the cythonized module:

```python
from distutils.core import setup
from Cython.Build import cythonize

setup(
ext_modules = cythonize("src/module.py")
)
```

then: `python setup.py build_ext --inplace`.

Note that using standard pickle works on both python & cythonized code, ie: `import pickle; pickle.dumps(my_obj.my_func)` works properly

Did I miss something?

I tried adding `__getstate__`/`__setstate__` that return `self.__dict__` but this does not solve the problem.

Does some of you have any clue about this? Thanks for the help :)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.