cloudpipe / cloudpipe/cloudpickle

Can't pickle/unpickle Cython cdef classes

Open
#186 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.9k
Forks
195
Avg merge
1d 10h
Merged PRs (30d)
1

Description

I'm not sure if this is a known limitations of cloud pickle that I was unaware of, but I haven't been able to get cloud pickle to pickle/unpickle instances of very basic cython-compiled classes.

If I have a TestClass.pyx file with the following content:

```
class TestClass(object):
pass
```

, then compile that to a TestClass.so file with cython and run the following:

```
import cloudpickle

from TestClass import TestClass

pickled_obj = cloudpickle.dumps(TestClass())
print cloudpickle.loads(pickled_obj)
```

, everything works just fine. However, if I change TestClass to a cdef class:

```
cdef class TestClass(object):
pass
```

, and recompile, I get the following error

```
Traceback (most recent call last):
File ".../Library/Preferences/PyCharm2017.2/scratches/scratch_2.py", line 6, in
print cloudpickle.loads(pickled_obj)
File ".../lib/python2.7/pickle.py", line 1388, in loads
return Unpickler(file).load()
File ".../lib/python2.7/pickle.py", line 864, in load
dispatch[key](self)
File ".../lib/python2.7/pickle.py", line 1139, in load_reduce
value = func(*args)
AttributeError: 'NoneType' object has no attribute '__pyx_unpickle_TestClass'
```

If I add any methods to this cdef class, like so:

```
cdef class TestClass(object):
cdef void my_fxn(self):
print "hello, world!"
```

, then I get the following error:

```
Traceback (most recent call last):
File ".../Library/Preferences/PyCharm2017.2/scratches/scratch_2.py", line 5, in
pickled_obj = cloudpickle.dumps(TestClass())
File ".../lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 895, in dumps
cp.dump(obj)
File ".../lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 268, in dump
return Pickler.dump(self, obj)
File ".../lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
File ".../lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File ".../lib/python2.7/pickle.py", line 401, in save_reduce
save(args)
File ".../lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File ".../lib/python2.7/pickle.py", line 554, in save_tuple
save(element)
File ".../tpounce/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File ".../lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 654, in save_global
return self.save_dynamic_class(obj)
File ".../lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 501, in save_dynamic_class
save(clsdict)
File ".../lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File ".../lib/python2.7/pickle.py", line 655, in save_dict
self._batch_setitems(obj.iteritems())
File ".../lib/python2.7/pickle.py", line 687, in _batch_setitems
save(v)
File ".../lib/python2.7/pickle.py", line 306, in save
rv = reduce(self.proto)
TypeError: can't pickle PyCapsule objects
```

Finally, if I add an __init__ method to my cdef class:

```
cdef class TestClass(object):
def __init__(self):
pass
```

, I get this error:

```
Traceback (most recent call last):
File ".../Library/Preferences/PyCharm2017.2/scratches/scratch_2.py", line 5, in
pickled_obj = cloudpickle.dumps(TestClass())
File ".../lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 895, in dumps
cp.dump(obj)
File ".../lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 268, in dump
return Pickler.dump(self, obj)
File ".../lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
File ".../lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File ".../lib/python2.7/pickle.py", line 401, in save_reduce
save(args)
File ".../lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File ".../lib/python2.7/pickle.py", line 554, in save_tuple
save(element)
File ".../lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File ".../lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 654, in save_global
return self.save_dynamic_class(obj)
File ".../lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 501, in save_dynamic_class
save(clsdict)
File ".../lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File ".../lib/python2.7/pickle.py", line 655, in save_dict
self._batch_setitems(obj.iteritems())
File ".../lib/python2.7/pickle.py", line 687, in _batch_setitems
save(v)
File ".../lib/python2.7/pickle.py", line 306, in save
rv = reduce(self.proto)
TypeError: can't pickle wrapper_descriptor objects
```

Are all of these just known limitations of cloudpickle, or should this be working? I am using:
cloudpickle==0.5.3
cython==0.28.3
python==2.7.13

Thanks!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.