huggingface / huggingface/datasets

load_dataset from s3 file system through streaming can't not iterate data

Open
#5,880 4 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

I have a JSON file in my s3 file system(minio), I can use load_dataset to get the file link, but I can't iterate it
image
image

we can change 4 lines to fix this bug, you can check whether it is ok for us.
image

### Steps to reproduce the bug

1. storage a file in you s3 file system
2. use load_dataset to read it through streaming
3. iterate it

### Expected behavior

can iterate it successfully

### Environment info

- `datasets` version: 2.12.0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.16
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.1

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.