huggingface / huggingface/datasets

[feature request] adding easy to remember `datasets.cache_dataset()` + `datasets.is_dataset_cached()`

Open
#2,845 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

Often, there is a need to prepare a dataset but not use it immediately, e.g. think tests suite setup, so it'd be really useful to be able to do:

```
if not datasets.is_dataset_cached(ds): datasets.cache_dataset(ds)
```

This can already be done with:
```
builder = load_dataset_builder(ds)
if not os.path.idsir(builder.cache_dir):
builder.download_and_prepare()
```

but the current way is a way less intuitive and much harder to remember than the proposed API, IMHO.

One more way is to do:

```
_ = load_dataset(ds)
```
but it wastes resources loading the dataset when it's not needed.

this has been discussed at https://huggingface.slack.com/archives/C01229B19EX/p1630021912025800

Thank you!

@lhoestq

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.