apache / apache/arrow

[C++][Python] Pass extra kwargs to S3 Pyarrow Filesystem

Open
#34,253 4 comments 1 reaction 0 assignees View on GitHub
Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

AFAIK current implementation of [Pyarrow s3fs](https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html) does not support passing extra kwargs to the various APIs. One example is [`x-amz-request-payer`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#API_ListObjectsV2_RequestSyntax) (i.e. Requester pays) which cannot be set in `get_file_info` or `open_input_file`.

#10251 has addressed this for one particular use case (ACL) but a more general approach would be preferred

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Start with Pyarrow's S3FileSystem API, especially get_file_info and open_input_file, and compare the existing ACL handling from #10251. Trace the C++ and Python interfaces to determine where extra request kwargs would enter; done means a general mechanism supports x-amz-request-payer across the relevant APIs.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, cpp, python
Domain
backend-api-design, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.