cloudpipe / cloudpipe/cloudpickle

Cannot exclude members to pickle in metaclass

Aperta
#482 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub
pickle issue
Lingua principale
Python
Stelle
1.9k
Fork
195
Merge medio
1g 10h
PR unite (30g)
1

Descrizione

In the example below, I implemented `__getstate__(cls)` that returns empty dict
in order not to pickle `cls.INSTANCES` and `cls.LOCK` members of metaclass `SingletonMeta`.

But it seems that `cloudpickle` still tries to pickle those members
and fails to pickle since `threading.RLock` is not picklable.

```python
import cloudpickle

from threading import RLock

class SingletonMeta(type):
def __new__(cls, name, bases, namespace):
return super().__new__(cls, name, bases, namespace)

def __init__(cls, name, bases, namespace):
super().__init__(name, bases, namespace)
cls.INSTANCES = {}
cls.LOCK = RLock()

def __call__(cls, *args, **kwargs):
key = cls.__getkey__(*args, **kwargs)
key = tuple(key.items())
if key not in cls.INSTANCES:
with cls.LOCK:
if key not in cls.INSTANCES:
cls.INSTANCES[key] = super().__call__(*args, **kwargs)
instance = cls.INSTANCES[key]
return instance

def __getstate__(cls):
# don't want to pickle cls.INSTANCES and cls.LOCK
return {}

def __setstate__(cls, state):
pass

class SingletonClass(metaclass=SingletonMeta):
def __init__(self, key):
self.key = key
self.transient = [self.key]

def __getstate__(self):
state = self.__dict__.copy()
del state["transient"]
return state

def __setstate__(self, state):
for name, value in state.items():
setattr(self, name, value)
self.transient = [self.key]

@classmethod
def __getkey__(cls, key):
return {"key": key}

# singleton implementation works fine as long as metaclass = SingletonMeta
def test_singleton():
singleton_instance1 = SingletonClass(0)
singleton_instance2 = SingletonClass(0)
singleton_instance3 = SingletonClass(1)
assert singleton_instance1 is singleton_instance2
assert singleton_instance1 is not singleton_instance3

# pickling instances works fine if metaclass != SingletonMeta
# but raises `TypeError: cannot pickle '_thread.RLock' object` if metaclass = SingletonMeta
def test_singleton_pickle():
singleton_instance1 = SingletonClass(0)
singleton_instance1_loaded = cloudpickle.loads(
cloudpickle.dumps(singleton_instance1)
)
assert singleton_instance1 is not singleton_instance1_loaded
assert singleton_instance1.key == singleton_instance1_loaded.key
assert singleton_instance1.transient == singleton_instance1_loaded.transient

if __name__ == "__main__":
# singleton implementation works fine as long as metaclass = SingletonMeta
test_singleton()

# pickling instances works fine if metaclass != SingletonMeta
# but raises `TypeError: cannot pickle '_thread.RLock' object` if metaclass = SingletonMeta
test_singleton_pickle()

# raises "TypeError: cannot pickle '_thread.RLock' object"
cloudpickle.dumps(test_singleton)
```

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Inizia riproducendo il fallimento nell’esempio SingletonMeta fornito, concentrandoti su test_singleton_pickle e sulla chiamata finale cloudpickle.dumps(test_singleton). Leggi come cloudpickle gestisce le metaclassi e i metodi __getstate__, quindi verifica che le istanze possano essere serializzate e deserializzate senza tentare di serializzare l’RLock della metaclasse, preservando le assertions mostrate.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python
Ambito
tooling
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.