[Python] pyarrow: using role_arn with S3FileSystem results in Anonymous user error
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
Hi, I'm initializing S3FileSystem with a role_arn, to get refreshable temporary credentials, as described in
https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html
The init call is the following:
```
self._fs = S3FileSystem(
region=_AWS_REGION,
role_arn=self._role_arn,
anonymous=False,
)
```
A bit later I'm initializing a parquetwriter with the file system
```
fs = _get_service().get_file_system()
output_url = output_url.replace("s3://", "")
schema = Schema.from_pandas(df=data)
writer = ParquetWriter(where=output_url, schema=schema, filesystem=fs)
```
I then get the error:
OSError: When initiating multiple part upload for key 'resources/01HBTZN7DBR1BE62S5TJRQDZXP/data.parquet' in bucket 'my-bucket-name': AWS Error ACCESS_DENIED during CreateMultipartUpload operation: Anonymous users cannot initiate multipart uploads. Please authenticate.
I expect that S3FileSystem would use STS to get temporary credentials like described in the documentation. The role is configured to allow the service (glue) to assume the role.
Pyarrow version 12.0.0
S3FS 2023.6.0
Using Pyarrow in a aws glue v4 environment
### Component(s)
Python
Contributor guide
Research direction
Start with the S3FileSystem initialization using role_arn and the subsequent ParquetWriter filesystem handoff described in the issue. Reproduce the multipart upload in the stated AWS Glue v4 environment and inspect how authentication is carried into CreateMultipartUpload. Done means identifying and addressing the Anonymous user behavior, with a regression test or documented limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100