apache / apache/arrow

[Python] write_to_parquet with s3fs will fails on no permissions for CreateBucket even when bucket exists (regression in 8.0.0)

Open
#32,974 2 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

Hello

in version 8.0.0 and later the following bug was introduced: `write_to_dataset` will fail with the error "An error occurred (AccessDenied) when calling the CreateBucket operation: Access Denied" on writing to existing S3 bucket.
```java

import s3fs
import pyarrow as pa
import pyarrow.parquet as pq
import pandas as pd

print(pa.__version__)

dff = pd.DataFrame({'Par1': ['a','b','c']*8, 'Par2': ['x','y']*12, 'Par3': range(24)})
path = 's3://bucket/path'

s3 = s3fs.S3FileSystem()
table = pa.Table.from_pandas(dff)

# This will pass
pq.write_table(table, path + '/table.parquet', filesystem=s3)

# This will fail: An error occurred (AccessDenied) when calling the CreateBucket operation: Access Denied
pq.write_to_dataset(table, path + '/table_partitioned.parquet', filesystem=s3, partition_cols=['Par1', 'Par2'])
```
In 7.0.0 the code runs without errors.

Best regards

Vladimir

**Reporter**: [Vladimir](https://issues.apache.org/jira/browse/ARROW-17739)
**Watchers**: [Rok Mihevc](https://issues.apache.org/jira/browse/ARROW-17739) / @rok

**Note**: *This issue was originally created as [ARROW-17739](https://issues.apache.org/jira/browse/ARROW-17739). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Reproduce the difference between versions 7.0.0 and 8.0.0 using the shown write_table and write_to_dataset calls with s3fs and an existing S3 bucket. Trace the Python Parquet dataset-writing entry point and its bucket handling, then add a regression test showing that partitioned writes do not require CreateBucket permission when the bucket already exists.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, pandas, python
Domain
cloud, data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.