cloudpipe / cloudpipe/cloudpickle
Pickling functions breaks if Python module defined from types.ModuleType is used
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 195
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 1
Description
Hi there!
Our project ([the Python bindings of ROOT](https://root.cern/)) uses inside the Python module a "facade" object to do some magic with the lookups in the module and this causes issues with cloudpickle, which shows up when using dask. However, this facade object, derived from a `types.ModuleType`, has never caused issues, also when using the standard pickle in the Python standard library.
Below you can find a standalone reproducer, which shows that the standard pickle is fine with such a construct but cloudpickle fails with the error message `TypeError: cannot pickle 'Facade' object`.
Edit: We have found out that we can move the `import my_module` line inside the function `foo` as a workaround. I hope this gives an hint to the underlying issue!
```python
# Define the module facade and inject it as module
import sys
from types import ModuleType
class Facade(ModuleType):
def __init__(self, name):
super().__init__(name)
self.foo = 42
sys.modules['my_module'] = Facade('my_module')
# Pickle a function which uses the module with the facade
import my_module
import cloudpickle as pickle # breaks with cloudpickle
#import pickle # works fine with the standard pickle
def foo():
my_module.foo
return 42
with open('f.pkl', 'wb') as f:
pickle.dump(foo, f) # Breaks with "TypeError: cannot pickle 'Facade' object"
```
Python: 3.8.6
cloudpickle: 1.6.0
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.