[Python] write_to_parquet with s3fs will fails on no permissions for CreateBucket even when bucket exists (regression in 8.0.0)
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
Hello
in version 8.0.0 and later the following bug was introduced: `write_to_dataset` will fail with the error "An error occurred (AccessDenied) when calling the CreateBucket operation: Access Denied" on writing to existing S3 bucket.
```java
import s3fs
import pyarrow as pa
import pyarrow.parquet as pq
import pandas as pd
print(pa.__version__)
dff = pd.DataFrame({'Par1': ['a','b','c']*8, 'Par2': ['x','y']*12, 'Par3': range(24)})
path = 's3://bucket/path'
s3 = s3fs.S3FileSystem()
table = pa.Table.from_pandas(dff)
# This will pass
pq.write_table(table, path + '/table.parquet', filesystem=s3)
# This will fail: An error occurred (AccessDenied) when calling the CreateBucket operation: Access Denied
pq.write_to_dataset(table, path + '/table_partitioned.parquet', filesystem=s3, partition_cols=['Par1', 'Par2'])
```
In 7.0.0 the code runs without errors.
Best regards
Vladimir
**Reporter**: [Vladimir](https://issues.apache.org/jira/browse/ARROW-17739)
**Watchers**: [Rok Mihevc](https://issues.apache.org/jira/browse/ARROW-17739) / @rok
**Note**: *This issue was originally created as [ARROW-17739](https://issues.apache.org/jira/browse/ARROW-17739). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Reproduce the difference between versions 7.0.0 and 8.0.0 using the shown write_table and write_to_dataset calls with s3fs and an existing S3 bucket. Trace the Python Parquet dataset-writing entry point and its bucket handling, then add a regression test showing that partitioned writes do not require CreateBucket permission when the bucket already exists.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, pandas, python
- Domain
- cloud, data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100