huggingface / huggingface/datasets

Provide a way to write a streamed dataset to the disk

Open
#3,142 2 comments 3 reactions 0 assignees View on GitHub
dataset-viewer enhancement
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

**Is your feature request related to a problem? Please describe.**

The streaming mode allows to get the 100 first rows of a dataset very quickly. But it does not cache the answer, so a posterior call to get the same 100 rows will send a request to the server again and again.

**Describe the solution you'd like**

Provide a way to write the streamed rows of a dataset on the disk, and to load from it later.

**Describe alternatives you've considered**

Provide a third mode: `lazy`, which would use the local cache for the data that have already been fetched previously, and use streaming to get the rest of the requested data.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.