huggingface / huggingface/datasets
Feature Request / Discussion: Skip origin_metadata fetch when streaming=True?
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
Hi team!
I've been looking into the initialization process of the `DatasetBuilder` and noticed that the ETag/origin metadata caching path is executed unconditionally.
While fetching the `origin_metadata` and calculating the ETag is necessary for safely managing a local disk cache when using `as_dataset()`, it seems this might be redundant for streaming. When a user requests `streaming=True`, the pipeline drops into `as_streaming_dataset()`. Because streaming does not use a local cache, it appears the streaming pipeline ignores the generated `_cache_dir` string and never calls `_get_dataset_fingerprint()`. As a result, the origin metadata is fetched and hashed without being used.
I wanted to start a discussion to understand if there is another purpose for the etag/origin_metadata when streaming. If not, would it be possible/desirable to skip this fetch when `streaming=True` to improve initialization latency and save unnecessary network calls?
Thanks!
Contributor guide
Assessment
This issue has not been assessed yet.