cloudpipe / cloudpipe/cloudpickle

Pickling functions breaks if Python module defined from types.ModuleType is used

Open
#397 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.9k
Forks
195
Avg merge
1d 10h
Merged PRs (30d)
1

Description

Hi there!

Our project ([the Python bindings of ROOT](https://root.cern/)) uses inside the Python module a "facade" object to do some magic with the lookups in the module and this causes issues with cloudpickle, which shows up when using dask. However, this facade object, derived from a `types.ModuleType`, has never caused issues, also when using the standard pickle in the Python standard library.

Below you can find a standalone reproducer, which shows that the standard pickle is fine with such a construct but cloudpickle fails with the error message `TypeError: cannot pickle 'Facade' object`.

Edit: We have found out that we can move the `import my_module` line inside the function `foo` as a workaround. I hope this gives an hint to the underlying issue!

```python
# Define the module facade and inject it as module

import sys
from types import ModuleType

class Facade(ModuleType):
def __init__(self, name):
super().__init__(name)
self.foo = 42

sys.modules['my_module'] = Facade('my_module')

# Pickle a function which uses the module with the facade

import my_module

import cloudpickle as pickle # breaks with cloudpickle
#import pickle # works fine with the standard pickle

def foo():
my_module.foo
return 42

with open('f.pkl', 'wb') as f:
pickle.dump(foo, f) # Breaks with "TypeError: cannot pickle 'Facade' object"
```

Python: 3.8.6
cloudpickle: 1.6.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.