s3fs fetching data from s3 fails while boto3 get_object for the same object works
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1k
- Forks
- 305
- Avg merge
- 22h 37m
- Merged PRs (30d)
- 4
Description
Env:
❯ ipython
Python 3.11.1 (main, Dec 23 2022, 09:28:24) [Clang 14.0.0 (clang-1400.0.29.202)]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.8.0 -- An enhanced Interactive Python. Type '?' for help.
Version:
In [1]: import s3fs
In [2]: s3fs.__version__
Out[2]: '2022.11.0'
In [3]: fsspec.__version__
Out[13]: '2022.11.0'
In [4]: import boto3
In [5]: boto3.setup_default_session(profile_name='datadeft-dev-eu')
In [7]: import fsspec
In [8]: with fsspec.open("s3://hellohexa-ci/iris.csv") as f:
...: print(f)
...:
.....
FileNotFoundError: hellohexa-ci/iris.csv
In [10]: s3 = boto3.client("s3")
In [11]: s3.get_object(Bucket="hellohexa-ci", Key="iris.csv").get("ResponseMetadata")
Out[11]:
{'RequestId': 'YJ04NEX2C5E9HPFV',
'HostId': '8OxPPB4ZFufC+EpSDxnM5o/DJ/SgOHb+bxMXk8Inb4MaWHzAHYZXgxrVPs9E2p4hHhLub+3eVIM=',
'HTTPStatusCode': 200,
'HTTPHeaders': {'x-amz-id-2': '8OxPPB4ZFufC+EpSDxnM5o/DJ/SgOHb+bxMXk8Inb4MaWHzAHYZXgxrVPs9E2p4hHhLub+3eVIM=',
'x-amz-request-id': 'YJ04NEX2C5E9HPFV',
'date': 'Sun, 08 Jan 2023 14:29:42 GMT',
'last-modified': 'Sat, 07 Jan 2023 20:55:47 GMT',
'etag': '"57fce90c81521889c736445f058c4838"',
'accept-ranges': 'bytes',
'content-type': 'text/csv',
'server': 'AmazonS3',
'content-length': '3716'},
'RetryAttempts': 0}
Not sure why this is happening.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with fsspec.open("s3://hellohexa-ci/iris.csv") and compare it with the successful boto3 get_object call using the same session. Read the S3 filesystem path and credential-handling flow first; done means the object opens through fsspec while the existing boto3 access continues to work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100