huggingface / huggingface/datasets

Load a cached dataset as iterable

Open
#5,481 22 comments 5 reactions 0 assignees View on GitHub
enhancement good second issue
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

The idea would be to allow something like

```python
ds = load_dataset("c4", "en", as_iterable=True)
```

To be used to train models. It would load an IterableDataset from the cached Arrow files.

Cc @stas00

Edit : from the discussions we may load from cache when streaming=True

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.