fsspec / fsspec/s3fs

s3fs 0.5.x incompatibility with Python 3.6

Open
#387 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
305
Avg merge
22h 37m
Merged PRs (30d)
4

Description

What happened:
Using s3fs==0.5.1 and python 3.6.12 is not possible to push data on S3 using pandas 1.1.3. The function asyncio.current_task is only available since python 3.7 (see https://docs.python.org/3/library/asyncio-task.html#asyncio.current_task).

To be able to push data on S3, I had to downgrade to s3fs==0.4.2.

What you expected to happen:
When I call pip install s3fs I expected it selects a version compatible with Python 3.6.

Could you fix with a workaround on asyncio.current_task or specify in setup.py that requires Python >=3.7, please?

Minimal Complete Verifiable Example:

>>> import pandas
>>> df = ...
>>> df.to_csv("s3://.../file.csv", index=False)
...
/usr/local/anaconda3/envs/wonderful/lib/python3.6/site-packages/fsspec/asyn.py in _run_until_done(coro)
     19     loop = asyncio.get_event_loop()
     20     assert loop.is_running()
---> 21     task = asyncio.current_task()
     22     asyncio.tasks._unregister_task(task)
     23     current_task = asyncio.tasks._current_tasks.get(loop)

AttributeError: module 'asyncio' has no attribute 'current_task'

Anything else we need to know?:

Using s3fs==0.5.0 I had another unexpected behavior and we were not able to specifiy the aws region at the pandas.
Furthermore this is a regression compare to the 0.4.2 version.

>>> import pandas
>>> df = ...
>>> df.to_csv("s3://.../file.csv", index=False)
...
/usr/local/anaconda3/envs/wonderful/lib/python3.6/site-packages/s3fs/core.py in _mkdir(self, path, acl, create_parents, **kwargs)
    448                 self.invalidate_cache(bucket)
    449             except ClientError as e:
--> 450                 raise translate_boto_error(e) from e
    451             except ParamValidationError as e:
    452                 raise ValueError('Bucket create failed %r: %s' % (bucket, e))

PermissionError: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to.

Environment:

  • Dask version: no used
  • Python version: python 3.6
  • Operating System: mac os x
  • Install method (conda, pip, source):
    • conda 4.8.3
    • pip 20.2.3
    • s3fs 0.5.1
    • pandas 1.1.3

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the Python 3.6 traceback from fsspec/asyn.py, then inspect setup.py for the supported Python versions. Review the related s3fs/core.py region error as a separate regression, and verify that both S3 uploads and region-specific bucket creation behave correctly on the supported versions.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, pandas, python
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.