fsspec / fsspec/s3fs

Writing to a cross-environment bucket requires CreateBucket permission

Open
#616 9 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
305
Avg merge
22h 37m
Merged PRs (30d)
4

Description

similar to #451, writing to a cross-account bucket tries to create the bucket which leads to an Access Denied error when the process doesn't have a CreateBucket permission. Because the least privilege is the best practice, this can happen in production for most of the cross-account s3 access. here's the stack trace:

>>> import pandas as pd
>>> import boto3
>>> import s3fs
>>> pd.__version__
'1.3.5'
>>> boto3.__version__
'1.21.21'
>>> s3fs.__version__
'2022.3.0'
>>> df = pd.DataFrame(range(10))
>>> df.to_csv('s3://d1g1t-dataloader-us/test1.txt', storage_options={"s3_additional_kwargs": {"ACL": "bucket-owner-full-control"}})
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/s3fs/core.py", line 282, in _call_s3
    out = await method(**additional_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/aiobotocore/client.py", line 228, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the CreateBucket operation: Access Denied

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/pandas/core/generic.py", line 3466, in to_csv
    return DataFrameRenderer(formatter).to_csv(
  File "/usr/local/lib/python3.8/dist-packages/pandas/io/formats/format.py", line 1105, in to_csv
    csv_formatter.save()
  File "/usr/local/lib/python3.8/dist-packages/pandas/io/formats/csvs.py", line 237, in save
    with get_handle(
  File "/usr/local/lib/python3.8/dist-packages/pandas/io/common.py", line 609, in get_handle
    ioargs = _get_filepath_or_buffer(
  File "/usr/local/lib/python3.8/dist-packages/pandas/io/common.py", line 369, in _get_filepath_or_buffer
    file_obj = fsspec.open(
  File "/usr/local/lib/python3.8/dist-packages/fsspec/core.py", line 462, in open
    return open_files(
  File "/usr/local/lib/python3.8/dist-packages/fsspec/core.py", line 305, in open_files
    [fs.makedirs(parent, exist_ok=True) for parent in parents]
  File "/usr/local/lib/python3.8/dist-packages/fsspec/core.py", line 305, in <listcomp>
    [fs.makedirs(parent, exist_ok=True) for parent in parents]
  File "/usr/local/lib/python3.8/dist-packages/fsspec/asyn.py", line 85, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/fsspec/asyn.py", line 65, in sync
    raise return_result
  File "/usr/local/lib/python3.8/dist-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/usr/local/lib/python3.8/dist-packages/s3fs/core.py", line 767, in _makedirs
    await self._mkdir(path, create_parents=True)
  File "/usr/local/lib/python3.8/dist-packages/s3fs/core.py", line 752, in _mkdir
    await self._call_s3("create_bucket", **params)
  File "/usr/local/lib/python3.8/dist-packages/s3fs/core.py", line 302, in _call_s3
    raise err
PermissionError: Access Denied

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the shown pandas-to-S3 call with the listed pandas, boto3, and s3fs versions, then inspect s3fs/core.py at _makedirs, _mkdir, and _call_s3. Compare the behavior with related issue #451 and verify that writing to an existing cross-account bucket no longer requires CreateBucket permission.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.