fsspec / fsspec/s3fs

version 2023.3.0 issue

Open
#724 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
305
Avg merge
22h 37m
Merged PRs (30d)
4

Description

After upgrade sf3 and fsspectp 2023.3.0 we see the following error, is there a bug or can someone advise how to resolve this? Thanks.

[ERROR] PermissionError: [Errno 1] The specified method is not allowed against this resource.
Traceback (most recent call last):
  File "/var/task/google_analytics/etl.py", line 119, in run_reports
    df.to_csv(get_s3_location(report), index=False)
  File "/var/task/pandas/core/generic.py", line 3551, in to_csv
    return DataFrameRenderer(formatter).to_csv(
  File "/var/task/pandas/io/formats/format.py", line 1180, in to_csv
    csv_formatter.save()
  File "/var/task/pandas/io/formats/csvs.py", line 241, in save
    with get_handle(
  File "/var/task/pandas/io/common.py", line 667, in get_handle
    ioargs = _get_filepath_or_buffer(
  File "/var/task/pandas/io/common.py", line 393, in _get_filepath_or_buffer
    file_obj = fsspec.open(
  File "/var/task/fsspec/core.py", line 419, in open
    return open_files(
  File "/var/task/fsspec/core.py", line 283, in open_files
    [fs.makedirs(parent, exist_ok=True) for parent in parents]
  File "/var/task/fsspec/core.py", line 283, in
    [fs.makedirs(parent, exist_ok=True) for parent in parents]
  File "/var/task/fsspec/asyn.py", line 115, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/var/task/fsspec/asyn.py", line 100, in sync
    raise return_result
  File "/var/task/fsspec/asyn.py", line 55, in _runner
    result[0] = await coro
  File "/var/task/s3fs/core.py", line 892, in _makedirs
    await self._mkdir(path, create_parents=True)
  File "/var/task/s3fs/core.py", line 877, in _mkdir
    await self._call_s3("create_bucket", **params)
  File "/var/task/s3fs/core.py", line 347, in _call_s3
    return await _error_wrapper(
  File "/var/task/s3fs/core.py", line 139, in _error_wrapper
    raise err

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with google_analytics/etl.py at line 119, then trace the pandas to_csv call through fsspec/core.py and s3fs/core.py, especially the _makedirs and _mkdir paths shown in the traceback. Reproduce the 2023.3.0 failure and determine whether the reported PermissionError is caused by the upgrade; done means the cause and a verified resolution are documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, pandas, python
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.