huggingface / huggingface/datasets

Directly reading parquet files in a s3 bucket from the load_dataset method

Open
#5,566 1 comment 0 reactions 0 assignees View on GitHub
duplicate enhancement
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Feature request

Right now, we have to read the get the parquet file to the local storage. So having ability to read given the bucket directly address would be benificial

### Motivation

In a production set up, this feature can help us a lot. So we do not need move training datafiles in between storage.

### Your contribution

I am willing to help if there's anyway.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.