huggingface / huggingface/datasets

Feature Request / Discussion: Skip origin_metadata fetch when streaming=True?

Open
#8,197 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

Hi team!

I've been looking into the initialization process of the `DatasetBuilder` and noticed that the ETag/origin metadata caching path is executed unconditionally.

While fetching the `origin_metadata` and calculating the ETag is necessary for safely managing a local disk cache when using `as_dataset()`, it seems this might be redundant for streaming. When a user requests `streaming=True`, the pipeline drops into `as_streaming_dataset()`. Because streaming does not use a local cache, it appears the streaming pipeline ignores the generated `_cache_dir` string and never calls `_get_dataset_fingerprint()`. As a result, the origin metadata is fetched and hashed without being used.

I wanted to start a discussion to understand if there is another purpose for the etag/origin_metadata when streaming. If not, would it be possible/desirable to skip this fetch when `streaming=True` to improve initialization latency and save unnecessary network calls?

Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.