Reading a webdataset from cloud storage
Open
@jantonguirao is already working on this.
Since Jan 17, 2025.
question
- Dominant language
- C++
- Stars
- 5.8k
- Forks
- 678
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 27
Description
Describe the question.
Hello,
I am curious how to read a webdataset from cloud storage (e.g. s3 or gcs)? I have tried using a pipe, providing the gs:// URI, and other options into the sample code provided here but cannot open the webdataset. Is there a way to do this, or do I need to combine the webdataset reader with another reader?
Check for duplicates
- I have searched the open bugs/issues and have found no duplicates for this bug report
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.