huggingface / huggingface/datasets
how to limit the size of memory mapped file?
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
Huggingface datasets use memory-mapped file to map large datasets in memory for fast access.
However, it seems like huggingface will occupy all the memory for memory-mapped files, which makes a troublesome situation since we cluster will distribute a small portion of memory to me (once it's over the limit, memory cannot be allocated), however, when the dataset checks the total memory, all of the memory will be taken into account which makes huggingface dataset try to allocate more memory than allowed.
So is there a way to explicitly limit the size of memory mapped file?
### Steps to reproduce the bug
python
>>> from datasets import load_dataset
>>> dataset = load_dataset("c4", "en", streaming=True)
### Expected behavior
In a normal environment, this will not have any problem.
However, when the system allocates a portion of the memory to the program and when the dataset checks the total memory, all of the memory will be taken into account which makes huggingface dataset try to allocate more memory than allowed.
### Environment info
linux cluster with SGE(Sun Grid Engine)
Contributor guide
Assessment
This issue has not been assessed yet.