cloudpipe / cloudpipe/cloudpickle

Cloudpickle non-deterministic dump when file is innocuously modified

Open
#385 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.9k
Forks
195
Avg merge
1d 10h
Merged PRs (30d)
1

Description

Cloudpickle seems to produce non-deterministic dumps when the file's formatting is "innocuously" modified (e.g., formatting changes outside of pickled object's definition) whereas `dill` and `pickle` would produce deterministic dumps.

For example, inserting a blank line anywhere after where the pickled function `foo` is defined will initially produce a different hash, then subsequently produce the same hash upon successive runs:
```
import cloudpickle
import dill
import pickle

def foo():
pass

def get_cpickle():
return cloudpickle.dumps(foo)

def get_dill():
return dill.dumps(foo)

def get_pickle():
return pickle.dumps(foo)

if __name__ == '__main__':
print('Cpickle:', hash(get_cpickle()))
print('Dill:', hash(get_dill()))
print('Pickle:', hash(get_pickle()))
```
Command:
```
PYTHONHASHSEED=1 python bad_pickle.py
```

First run:
```
Cpickle: -185195056977094428
Dill: 1827482599472099751
Pickle: -2221802750934099445
```
Second run:
```
Cpickle: 5072829361071368526
Dill: 1827482599472099751
Pickle: -2221802750934099445
```
Blank line inserted after `print('Cpickle:', ...)` (third run):
```
Cpickle: -185195056977094428
Dill: 1827482599472099751
Pickle: -2221802750934099445
```
Fourth run:
```
Cpickle: 5072829361071368526
Dill: 1827482599472099751
Pickle: -2221802750934099445
```

This was tested on the following versions:
```
Cpickle version: 1.2.2
Dill version: 0.2.7.1
Python version: 3.6.10 (default, Jan 1 2020, 00:00:00)
```

This seems like perhaps Cloudpickle is also hashing some eventually cached version of the source file (e.g., .pyc).

This is also somewhat related to #120 .

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.