s3fs 0.5.x incompatibility with Python 3.6
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1k
- Forks
- 305
- Avg merge
- 22h 37m
- Merged PRs (30d)
- 4
Description
What happened:
Using s3fs==0.5.1 and python 3.6.12 is not possible to push data on S3 using pandas 1.1.3. The function asyncio.current_task is only available since python 3.7 (see https://docs.python.org/3/library/asyncio-task.html#asyncio.current_task).
To be able to push data on S3, I had to downgrade to s3fs==0.4.2.
What you expected to happen:
When I call pip install s3fs I expected it selects a version compatible with Python 3.6.
Could you fix with a workaround on asyncio.current_task or specify in setup.py that requires Python >=3.7, please?
Minimal Complete Verifiable Example:
>>> import pandas
>>> df = ...
>>> df.to_csv("s3://.../file.csv", index=False)
...
/usr/local/anaconda3/envs/wonderful/lib/python3.6/site-packages/fsspec/asyn.py in _run_until_done(coro)
19 loop = asyncio.get_event_loop()
20 assert loop.is_running()
---> 21 task = asyncio.current_task()
22 asyncio.tasks._unregister_task(task)
23 current_task = asyncio.tasks._current_tasks.get(loop)
AttributeError: module 'asyncio' has no attribute 'current_task'
Anything else we need to know?:
Using s3fs==0.5.0 I had another unexpected behavior and we were not able to specifiy the aws region at the pandas.
Furthermore this is a regression compare to the 0.4.2 version.
>>> import pandas
>>> df = ...
>>> df.to_csv("s3://.../file.csv", index=False)
...
/usr/local/anaconda3/envs/wonderful/lib/python3.6/site-packages/s3fs/core.py in _mkdir(self, path, acl, create_parents, **kwargs)
448 self.invalidate_cache(bucket)
449 except ClientError as e:
--> 450 raise translate_boto_error(e) from e
451 except ParamValidationError as e:
452 raise ValueError('Bucket create failed %r: %s' % (bucket, e))
PermissionError: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to.
Environment:
- Dask version: no used
- Python version: python 3.6
- Operating System: mac os x
- Install method (conda, pip, source):
- conda 4.8.3
- pip 20.2.3
- s3fs 0.5.1
- pandas 1.1.3
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the Python 3.6 traceback from fsspec/asyn.py, then inspect setup.py for the supported Python versions. Review the related s3fs/core.py region error as a separate regression, and verify that both S3 uploads and region-specific bucket creation behave correctly on the supported versions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, pandas, python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100