huggingface / huggingface/datasets
Add the option of saving in parquet instead of arrow
Open
enhancement
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Feature request
In dataset.save_to_disk('/path/to/save/dataset'),
add the option to save in parquet format
dataset.save_to_disk('/path/to/save/dataset', format="parquet"),
because arrow is not used for Production Big data.... (only parquet)
### Motivation
because arrow is not used for Production Big data.... (only parquet)
### Your contribution
I can do the testing !
Contributor guide
Assessment
This issue has not been assessed yet.