huggingface / huggingface/datasets
load_from_disk
Open
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
I have data saved with save_to_disk. The data is big (700Gb). When I try loading it, the only option is load_from_disk, and this function copies the data to a tmp directory, causing me to run out of disk space. Is there an alternative solution to that?
### Steps to reproduce the bug
when trying to load data using load_From_disk after being saved using save_to_disk
### Expected behavior
run out of disk space
### Environment info
lateest version
Contributor guide
Assessment
This issue has not been assessed yet.