[C++][Python] Pass extra kwargs to S3 Pyarrow Filesystem
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
AFAIK current implementation of [Pyarrow s3fs](https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html) does not support passing extra kwargs to the various APIs. One example is [`x-amz-request-payer`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#API_ListObjectsV2_RequestSyntax) (i.e. Requester pays) which cannot be set in `get_file_info` or `open_input_file`.
#10251 has addressed this for one particular use case (ACL) but a more general approach would be preferred
### Component(s)
Python
Contributor guide
Research direction
Start with Pyarrow's S3FileSystem API, especially get_file_info and open_input_file, and compare the existing ACL handling from #10251. Trace the C++ and Python interfaces to determine where extra request kwargs would enter; done means a general mechanism supports x-amz-request-payer across the relevant APIs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, cpp, python
- Domain
- backend-api-design, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100