Writing to a cross-environment bucket requires CreateBucket permission
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1k
- Forks
- 305
- Avg merge
- 22h 37m
- Merged PRs (30d)
- 4
Description
similar to #451, writing to a cross-account bucket tries to create the bucket which leads to an Access Denied error when the process doesn't have a CreateBucket permission. Because the least privilege is the best practice, this can happen in production for most of the cross-account s3 access. here's the stack trace:
>>> import pandas as pd
>>> import boto3
>>> import s3fs
>>> pd.__version__
'1.3.5'
>>> boto3.__version__
'1.21.21'
>>> s3fs.__version__
'2022.3.0'
>>> df = pd.DataFrame(range(10))
>>> df.to_csv('s3://d1g1t-dataloader-us/test1.txt', storage_options={"s3_additional_kwargs": {"ACL": "bucket-owner-full-control"}})
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/s3fs/core.py", line 282, in _call_s3
out = await method(**additional_kwargs)
File "/usr/local/lib/python3.8/dist-packages/aiobotocore/client.py", line 228, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the CreateBucket operation: Access Denied
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 3466, in to_csv
return DataFrameRenderer(formatter).to_csv(
File "/usr/local/lib/python3.8/dist-packages/pandas/io/formats/format.py", line 1105, in to_csv
csv_formatter.save()
File "/usr/local/lib/python3.8/dist-packages/pandas/io/formats/csvs.py", line 237, in save
with get_handle(
File "/usr/local/lib/python3.8/dist-packages/pandas/io/common.py", line 609, in get_handle
ioargs = _get_filepath_or_buffer(
File "/usr/local/lib/python3.8/dist-packages/pandas/io/common.py", line 369, in _get_filepath_or_buffer
file_obj = fsspec.open(
File "/usr/local/lib/python3.8/dist-packages/fsspec/core.py", line 462, in open
return open_files(
File "/usr/local/lib/python3.8/dist-packages/fsspec/core.py", line 305, in open_files
[fs.makedirs(parent, exist_ok=True) for parent in parents]
File "/usr/local/lib/python3.8/dist-packages/fsspec/core.py", line 305, in <listcomp>
[fs.makedirs(parent, exist_ok=True) for parent in parents]
File "/usr/local/lib/python3.8/dist-packages/fsspec/asyn.py", line 85, in wrapper
return sync(self.loop, func, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/fsspec/asyn.py", line 65, in sync
raise return_result
File "/usr/local/lib/python3.8/dist-packages/fsspec/asyn.py", line 25, in _runner
result[0] = await coro
File "/usr/local/lib/python3.8/dist-packages/s3fs/core.py", line 767, in _makedirs
await self._mkdir(path, create_parents=True)
File "/usr/local/lib/python3.8/dist-packages/s3fs/core.py", line 752, in _mkdir
await self._call_s3("create_bucket", **params)
File "/usr/local/lib/python3.8/dist-packages/s3fs/core.py", line 302, in _call_s3
raise err
PermissionError: Access Denied
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the shown pandas-to-S3 call with the listed pandas, boto3, and s3fs versions, then inspect s3fs/core.py at _makedirs, _mkdir, and _call_s3. Compare the behavior with related issue #451 and verify that writing to an existing cross-account bucket no longer requires CreateBucket permission.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100