huggingface / huggingface/datasets
load_dataset on AWS lambda throws OSError(30, 'Read-only file system') error
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
I'm using AWS lambda to run a python application. I run the `load_dataset` function with cache_dir="/tmp" and is still throws the OSError(30, 'Read-only file system') error. Is even updated all the HF envs to point to /tmp dir but the issue still persists. I can confirm that the I can write to /tmp directory.
### Steps to reproduce the bug
```python
d = load_dataset(
path=hugging_face_link,
split=split,
token=token,
cache_dir="/tmp/hugging_face_cache",
)
```
### Expected behavior
Everything written to the file system as part of the load_datasets function should be in the /tmp directory.
### Environment info
datasets version: 2.16.1
Platform: Linux-5.10.216-225.855.amzn2.x86_64-x86_64-with-glibc2.26
Python version: 3.11.9
huggingface_hub version: 0.19.4
PyArrow version: 16.1.0
Pandas version: 2.2.2
fsspec version: 2023.10.0
Contributor guide
Assessment
This issue has not been assessed yet.