cloudpipe / cloudpipe/cloudpickle
Can't pickle/unpickle Cython cdef classes
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 195
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 1
Description
I'm not sure if this is a known limitations of cloud pickle that I was unaware of, but I haven't been able to get cloud pickle to pickle/unpickle instances of very basic cython-compiled classes.
If I have a TestClass.pyx file with the following content:
```
class TestClass(object):
pass
```
, then compile that to a TestClass.so file with cython and run the following:
```
import cloudpickle
from TestClass import TestClass
pickled_obj = cloudpickle.dumps(TestClass())
print cloudpickle.loads(pickled_obj)
```
, everything works just fine. However, if I change TestClass to a cdef class:
```
cdef class TestClass(object):
pass
```
, and recompile, I get the following error
```
Traceback (most recent call last):
File ".../Library/Preferences/PyCharm2017.2/scratches/scratch_2.py", line 6, in
print cloudpickle.loads(pickled_obj)
File ".../lib/python2.7/pickle.py", line 1388, in loads
return Unpickler(file).load()
File ".../lib/python2.7/pickle.py", line 864, in load
dispatch[key](self)
File ".../lib/python2.7/pickle.py", line 1139, in load_reduce
value = func(*args)
AttributeError: 'NoneType' object has no attribute '__pyx_unpickle_TestClass'
```
If I add any methods to this cdef class, like so:
```
cdef class TestClass(object):
cdef void my_fxn(self):
print "hello, world!"
```
, then I get the following error:
```
Traceback (most recent call last):
File ".../Library/Preferences/PyCharm2017.2/scratches/scratch_2.py", line 5, in
pickled_obj = cloudpickle.dumps(TestClass())
File ".../lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 895, in dumps
cp.dump(obj)
File ".../lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 268, in dump
return Pickler.dump(self, obj)
File ".../lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
File ".../lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File ".../lib/python2.7/pickle.py", line 401, in save_reduce
save(args)
File ".../lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File ".../lib/python2.7/pickle.py", line 554, in save_tuple
save(element)
File ".../tpounce/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File ".../lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 654, in save_global
return self.save_dynamic_class(obj)
File ".../lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 501, in save_dynamic_class
save(clsdict)
File ".../lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File ".../lib/python2.7/pickle.py", line 655, in save_dict
self._batch_setitems(obj.iteritems())
File ".../lib/python2.7/pickle.py", line 687, in _batch_setitems
save(v)
File ".../lib/python2.7/pickle.py", line 306, in save
rv = reduce(self.proto)
TypeError: can't pickle PyCapsule objects
```
Finally, if I add an __init__ method to my cdef class:
```
cdef class TestClass(object):
def __init__(self):
pass
```
, I get this error:
```
Traceback (most recent call last):
File ".../Library/Preferences/PyCharm2017.2/scratches/scratch_2.py", line 5, in
pickled_obj = cloudpickle.dumps(TestClass())
File ".../lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 895, in dumps
cp.dump(obj)
File ".../lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 268, in dump
return Pickler.dump(self, obj)
File ".../lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
File ".../lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File ".../lib/python2.7/pickle.py", line 401, in save_reduce
save(args)
File ".../lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File ".../lib/python2.7/pickle.py", line 554, in save_tuple
save(element)
File ".../lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File ".../lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 654, in save_global
return self.save_dynamic_class(obj)
File ".../lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 501, in save_dynamic_class
save(clsdict)
File ".../lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File ".../lib/python2.7/pickle.py", line 655, in save_dict
self._batch_setitems(obj.iteritems())
File ".../lib/python2.7/pickle.py", line 687, in _batch_setitems
save(v)
File ".../lib/python2.7/pickle.py", line 306, in save
rv = reduce(self.proto)
TypeError: can't pickle wrapper_descriptor objects
```
Are all of these just known limitations of cloudpickle, or should this be working? I am using:
cloudpickle==0.5.3
cython==0.28.3
python==2.7.13
Thanks!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.