huggingface / huggingface/datasets

Allow load_dataset to take a working dir for intermediate data

Open
#5,679 4 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Feature request

As a user, I can set a working dir for intermediate data creation. The processed files will be moved to the cache dir, like
```
load_dataset(…, working_dir=”/temp/dir”, cache_dir=”/cloud_dir”).
```

### Motivation

This will help the use case for using datasets with cloud storage as cache. It will help boost the performance.

### Your contribution

I can provide a PR to fix this if the proposal seems reasonable.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.