fsspec / fsspec/filesystem_spec
fsspec.asyn.sync shoud check the liveness of IO thread to avoid deadlocks
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
fsspec.asyn creates and runs an event loop used by async file system implementations as the default event loop.
However, this module does not explicitly close the event loop. As a result, when a Python interpreter enters the shutdown sequence, we experience a specific period during which the event loop is still marked as "running", but the IO thread running the loop has already stopped.
import fsspec.asyn
class Dummy:
def __init__(self):
self.loop = fsspec.asyn.get_loop()
def __del__(self):
print(fsspec.asyn.loop[0])
print(fsspec.asyn.iothread[0])
dummy = None
if __name__ == "__main__":
dummy = Dummy()
<_UnixSelectorEventLoop running=True closed=False debug=False> # Event loop is marked as "running"
<Thread(fsspecIO, stopped daemon 140283887175232)> # IO thread has stopped -> call of sync() with the loop will lead to a deadlock
This period is dangerous as it can lead to unexpected deadlocks, particularly if there are unclosed files that potentially trigger file system access when they are garbage-collected at the final moment of the interpreter shutdown, as I reported in #1685.
To mitigate this risk, we should add a liveness check of the IO thread in fsspec.asyn.sync() widely used by async file system implementations to run async functions synchronously.
def sync(loop, func, *args, timeout=None, **kwargs):
if loop is None or loop.is_closed():
raise RuntimeError("Loop is not running")
if iothread[0] and loop == get_loop() and not iothread[0].is_alive(): # New
raise RuntimeError("fsspec IO thread has already stopped.")
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with fsspec.asyn.sync() and inspect how get_loop() and iothread are used. Reproduce the interpreter-shutdown scenario described in the issue and verify that calling sync() after the IO thread stops raises the intended RuntimeError instead of deadlocking.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100