Problems with background threads
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 89
- Forks
- 21
- PR merge metrics
- No merged PRs in 30d
Description
I am trying to use this library with FUSE. However, I have noticed:
- The FUSE "fork into background" functionality results in unkillable I/O hangs because
SSHFileSystemseems to start background threads. I have this problem myself in rapidgzip with the thread pool, for which I added a method to join all threads, so that the process can be forked into the background, where the thread pool gets automatically restarted on first usage. It would be nice to have a similar API for SSHFilesystem. - When an error occurs, my program does not quit anymore when a SSHFilesystem was created. I have to press Ctrl+C / send SIGINT for it to quit. I guess that this is also because of background threads that do "block" somehow. Maybe I am just not closing something correctly...
The running threads according to import threading; threading.enumerate() are: MainThread, fsspecIO, asyncio_0. Maybe I am running into https://github.com/fsspec/sshfs/issues/42 Then again, looking at the code, it does not seem like there is any logic for joining or canceling that thread at all!?
On further inspection, the program also hangs inside sshfs.file.SSHFile.close when calling it manually, which itself hangs in SSHFile._close, which itself hangs in fsspec.asyn.sync_wrapper, which hangs in this loop.
It would already be helpful if SSHFile.close would forward kwargs like all the other methods that are initialized with __mirror_method, so that an argument with timeout = 3, for example, can be specified. Some kind of timeout should then also be used in __exit__.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with SSHFile.close and SSHFile._close in sshfs/file.py, then trace fsspec.asyn.sync_wrapper and its loop in fsspec/asyn.py while reproducing the reported background-thread and close hangs. Check the behavior of the fsspecIO and asyncio_0 threads, including the requested close kwargs and timeout path. Done means the reported hangs are addressed and close behavior is covered for the described usage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100