fsspec / fsspec/s3fs

Reading errors after upgrade from 0.2.0 to 0.4.0

Open
#271 20 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
305
Avg merge
22h 37m
Merged PRs (30d)
4

Description

Hi
We are running Flask service on a docker image.
one of our main routes reads file from S3 and returns it's content.
recently we upgraded the package from version 0.2.0 to 0.4.0.

After the upgrade, the route is working fine, but after several hours it raises strange exceptions contains only the file path (without any other information)

Our temporary solution for that is to reset the container, after that it's working fine, but fails again after several hours.

The most strange thing is that when I am connecting to the same container (with the connection issues) and I am opening python shell and execute the code - It's working as expected, even when the flask service on the same container raise exceptions on the same file.

any ideas?
thanks

my code:

fs = s3fs.S3FileSystem()
with fs.open('PATH_TO_FILE.json', 'rb') as f:
    data = f.read()
    f.close()
    json_content =  json.loads(data.decode('utf-8'))

print(json_content)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The reported entry point is the Flask route that reads PATH_TO_FILE.json from S3 through s3fs, with the Python shell comparison as a clue. Reproduce the failure after the 0.2.0-to-0.4.0 upgrade and capture the full exception while comparing route and shell behavior; done means identifying a reproducible cause and documenting the verified fix or regression.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, flask, python
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.