fsspec / fsspec/s3fs

Interpreter doesn't exist if s3 object is not closed

Open
#573 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
305
Avg merge
22h 37m
Merged PRs (30d)
4

Description

I have this absolutely minimal program:

import s3fs
s3 = s3fs.S3FileSystem().open("bucket/file", "wb")
print("END")

Taking a look at this, I would expect that if you start it with python program.py, it will print an END and exit. Instead it prints END and never exits.
If I put an s3.close() it works as expected.

Printing a stack trace after print gives:

Current thread 0x00007f102cfc2740 (most recent call first):
  File "/usr/local/lib/python3.10/threading.py", line 324 in wait
  File "/usr/local/lib/python3.10/threading.py", line 600 in wait
  File "/usr/local/lib/python3.10/site-packages/fsspec/asyn.py", line 59 in sync
  File "/usr/local/lib/python3.10/site-packages/fsspec/asyn.py", line 91 in wrapper
  File "/usr/local/lib/python3.10/site-packages/s3fs/core.py", line 1970 in commit
  File "/usr/local/lib/python3.10/site-packages/s3fs/core.py", line 1961 in _upload_chunk
  File "/usr/local/lib/python3.10/site-packages/fsspec/spec.py", line 1522 in flush
  File "/usr/local/lib/python3.10/site-packages/fsspec/spec.py", line 1651 in close
  File "/usr/local/lib/python3.10/site-packages/fsspec/spec.py", line 1673 in __del__

Packages:

$ pip freeze
aiobotocore==2.0.1
aiohttp==3.8.1
aioitertools==0.8.0
aiosignal==1.2.0
async-timeout==4.0.2
attrs==21.2.0
botocore==1.22.8
charset-normalizer==2.0.9
frozenlist==1.2.0
fsspec==2021.11.1
idna==3.3
jmespath==0.10.0
multidict==5.2.0
python-dateutil==2.8.2
s3fs==2021.11.1
six==1.16.0
urllib3==1.26.7
wrapt==1.13.3
yarl==1.7.2
$ python -V
Python 3.10.1

(but it fails with older python and module versions as well, for eg. we run 2021.8.1 in prod)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the minimal Python reproduction and the stack frames in fsspec/asyn.py, s3fs/core.py, and fsspec/spec.py, focusing on the upload and destructor paths. Check how an unclosed S3 object is finalized and whether the process can exit without an explicit close while preserving the expected upload behavior. Done means the reproduction prints END and exits normally.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.