apache / apache/arrow

[Python] pyarrow: using role_arn with S3FileSystem results in Anonymous user error

Open
#38,421 5 comments 1 reaction 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

Hi, I'm initializing S3FileSystem with a role_arn, to get refreshable temporary credentials, as described in
https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html

The init call is the following:

```
self._fs = S3FileSystem(
region=_AWS_REGION,
role_arn=self._role_arn,
anonymous=False,
)

```

A bit later I'm initializing a parquetwriter with the file system
```
fs = _get_service().get_file_system()
output_url = output_url.replace("s3://", "")
schema = Schema.from_pandas(df=data)
writer = ParquetWriter(where=output_url, schema=schema, filesystem=fs)
```

I then get the error:

OSError: When initiating multiple part upload for key 'resources/01HBTZN7DBR1BE62S5TJRQDZXP/data.parquet' in bucket 'my-bucket-name': AWS Error ACCESS_DENIED during CreateMultipartUpload operation: Anonymous users cannot initiate multipart uploads. Please authenticate.

I expect that S3FileSystem would use STS to get temporary credentials like described in the documentation. The role is configured to allow the service (glue) to assume the role.

Pyarrow version 12.0.0
S3FS 2023.6.0
Using Pyarrow in a aws glue v4 environment

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Start with the S3FileSystem initialization using role_arn and the subsequent ParquetWriter filesystem handoff described in the issue. Reproduce the multipart upload in the stated AWS Glue v4 environment and inspect how authentication is carried into CreateMultipartUpload. Done means identifying and addressing the Anonymous user behavior, with a regression test or documented limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.