Interpreter doesn't exist if s3 object is not closed
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1k
- Forks
- 305
- Avg merge
- 22h 37m
- Merged PRs (30d)
- 4
Description
I have this absolutely minimal program:
import s3fs
s3 = s3fs.S3FileSystem().open("bucket/file", "wb")
print("END")
Taking a look at this, I would expect that if you start it with python program.py, it will print an END and exit. Instead it prints END and never exits.
If I put an s3.close() it works as expected.
Printing a stack trace after print gives:
Current thread 0x00007f102cfc2740 (most recent call first):
File "/usr/local/lib/python3.10/threading.py", line 324 in wait
File "/usr/local/lib/python3.10/threading.py", line 600 in wait
File "/usr/local/lib/python3.10/site-packages/fsspec/asyn.py", line 59 in sync
File "/usr/local/lib/python3.10/site-packages/fsspec/asyn.py", line 91 in wrapper
File "/usr/local/lib/python3.10/site-packages/s3fs/core.py", line 1970 in commit
File "/usr/local/lib/python3.10/site-packages/s3fs/core.py", line 1961 in _upload_chunk
File "/usr/local/lib/python3.10/site-packages/fsspec/spec.py", line 1522 in flush
File "/usr/local/lib/python3.10/site-packages/fsspec/spec.py", line 1651 in close
File "/usr/local/lib/python3.10/site-packages/fsspec/spec.py", line 1673 in __del__
Packages:
$ pip freeze
aiobotocore==2.0.1
aiohttp==3.8.1
aioitertools==0.8.0
aiosignal==1.2.0
async-timeout==4.0.2
attrs==21.2.0
botocore==1.22.8
charset-normalizer==2.0.9
frozenlist==1.2.0
fsspec==2021.11.1
idna==3.3
jmespath==0.10.0
multidict==5.2.0
python-dateutil==2.8.2
s3fs==2021.11.1
six==1.16.0
urllib3==1.26.7
wrapt==1.13.3
yarl==1.7.2
$ python -V
Python 3.10.1
(but it fails with older python and module versions as well, for eg. we run 2021.8.1 in prod)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the minimal Python reproduction and the stack frames in fsspec/asyn.py, s3fs/core.py, and fsspec/spec.py, focusing on the upload and destructor paths. Check how an unclosed S3 object is finalized and whether the process can exit without an explicit close while preserving the expected upload behavior. Done means the reproduction prints END and exits normally.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100