huggingface / huggingface/datasets

load_dataset on AWS lambda throws OSError(30, 'Read-only file system') error

Open
#7,029 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

I'm using AWS lambda to run a python application. I run the `load_dataset` function with cache_dir="/tmp" and is still throws the OSError(30, 'Read-only file system') error. Is even updated all the HF envs to point to /tmp dir but the issue still persists. I can confirm that the I can write to /tmp directory.

### Steps to reproduce the bug

```python
d = load_dataset(
path=hugging_face_link,
split=split,
token=token,
cache_dir="/tmp/hugging_face_cache",
)
```

### Expected behavior

Everything written to the file system as part of the load_datasets function should be in the /tmp directory.

### Environment info

datasets version: 2.16.1
Platform: Linux-5.10.216-225.855.amzn2.x86_64-x86_64-with-glibc2.26
Python version: 3.11.9
huggingface_hub version: 0.19.4
PyArrow version: 16.1.0
Pandas version: 2.2.2
fsspec version: 2023.10.0

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.