meta-pytorch / meta-pytorch/data
Add support for `kwargs` in S3 DataPipes
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 179
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 2
Description
🚀 The feature
S3FileLister and S3FileLoader currently doesn't support keyword argument beyond request_timeout_ms, region, buffer_size, and multi_part_download. Th
One example is here, where a user would like to read a specify version of the bucket. I imagine there are similar parameters that users may want to pass through. Some of which may be passed to the construction of the S3Handler and some others may be used in handler.s3_read().
Motivation, pitch
This will allow users to specify additional parameters to interact with S3 according to their needs.
Alternatives
Add support in fsspec DataPipes instead and asks users to use those if necessary.
Additional context
No response
cc: @ejguan
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing S3FileLister and S3FileLoader into S3Handler, including handler.s3_read(), and review how the existing request_timeout_ms, region, buffer_size, and multi_part_download arguments are routed. Done means additional keyword arguments can reach the appropriate S3 construction or read operation, including selecting a specific bucket version, with tests covering the forwarding behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, data
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100