huggingface / huggingface/datasets
Support custom fingerprinting with `Dataset.from_generator`
Open
enhancement
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Feature request
When using `Dataset.from_generator`, the generator is hashed when building the fingerprint. Similar to `.map`, it would be interesting to let the user bypass this hashing by accepting a `fingerprint` argument to `.from_generator`.
### Motivation
Using the `.from_generator` constructor with a non-picklable generator fails. By accepting a `fingerprint` argument to `.from_generator`, the user would have the opportunity to manually fingerprint the dataset and thus bypass the crash.
### Your contribution
If validated, I can try to submit a PR for this.
Contributor guide
Assessment
This issue has not been assessed yet.