huggingface / huggingface/datasets
[feature request] adding easy to remember `datasets.cache_dataset()` + `datasets.is_dataset_cached()`
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
Often, there is a need to prepare a dataset but not use it immediately, e.g. think tests suite setup, so it'd be really useful to be able to do:
```
if not datasets.is_dataset_cached(ds): datasets.cache_dataset(ds)
```
This can already be done with:
```
builder = load_dataset_builder(ds)
if not os.path.idsir(builder.cache_dir):
builder.download_and_prepare()
```
but the current way is a way less intuitive and much harder to remember than the proposed API, IMHO.
One more way is to do:
```
_ = load_dataset(ds)
```
but it wastes resources loading the dataset when it's not needed.
this has been discussed at https://huggingface.slack.com/archives/C01229B19EX/p1630021912025800
Thank you!
@lhoestq
Contributor guide
Assessment
This issue has not been assessed yet.