huggingface / huggingface/datasets
Warnings and documentation about pickling incorrect
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
## Describe the bug
I have a docs bug and a closely related docs enhancement suggestion!
### Bug
The warning and documentation say "either `dill` or `pickle`" for fingerprinting. But it seems that `dill`, which is installed by `datasets` by default, _must_ work, or else the fingerprinting fails.
Warning:
https://github.com/huggingface/datasets/blob/450b9174765374111e5c6daab0ed294bc3d9b639/src/datasets/fingerprint.py#L262
Docs:
> For a transform to be hashable, it needs to be pickleable using dill or pickle.
> – [docs](https://huggingface.co/docs/datasets/processing.html#fingerprinting)
For my code, `pickle` works, but `dill` fails. The `dill` failure has already been reported in https://github.com/huggingface/datasets/issues/2643. However, the `dill` failure causes a hashing failure in the datasets library, without any backing off to `pickle`. This implies that it's not the case that either `dill` **or** `pickle` can work, but that `dill` must work if it is installed. I think this is more accurate wording, since it is installed and used by default:
https://github.com/huggingface/datasets/blob/c93525dc291346e54212567fa72d7d607befe937/setup.py#L83
... and the hashing will fail if it fails.
### Enhancement
I think it'd be very helpful to add to the documentation how to debug hashing failures. It took me a while to figure out how to diagnose this. There is a very nice two-liner by @lhoestq in https://github.com/huggingface/datasets/issues/2516#issuecomment-865173139:
```python
from datasets.fingerprint import Hasher
Hasher.hash(my_object)
```
I think add this to the docs will help future users quickly debug any hashing troubles of their own :-)
## Steps to reproduce the bug
`dill` but not `pickle` hashing failure in https://github.com/huggingface/datasets/issues/2643
## Expected results
If either `dill` or `pickle` can successfully hash, the hashing will succeed.
## Actual results
If `dill` or `pickle` cannot hash, the hashing fails.
## Environment info
- `datasets` version: 1.9.0
- Platform: Linux-5.8.0-1038-gcp-x86_64-with-glibc2.31
- Python version: 3.9.6
- PyArrow version: 4.0.1
Contributor guide
Assessment
This issue has not been assessed yet.