huggingface / huggingface/datasets

Allow dataset streaming from private a private source when loading a dataset with a dataset loading script

Open
#5,244 5 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Feature request

Add arguments to the function _get_authentication_headers_for_url_ like custom_endpoint and custom_token in order to add flexibility when downloading files from a private source.

It should also be possible to provide these arguments from the dataset loading script, maybe giving them to the dl_manager

### Motivation

It is possible to share a dataset hosted on another platform by writing a dataset loading script. It works perfectly for publicly available resources.

For resources that require authentication, you can provide a [download_custom](https://huggingface.co/docs/datasets/package_reference/builder_classes#datasets.DownloadManager) method to the download_manager.

Unfortunately, this function doesn't work with **dataset streaming**.

A solution so as to allow dataset streaming from private sources would be a more flexible _get_authentication_headers_for_url_ function.

### Your contribution

Would you be interested in this improvement ?
If so I could provide a PR. I've got something working locally, but it's not very clean, I'd need some guidance regarding integration.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.