[Feature Request] Fork Safety for async filesystems

Open
#835 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
aws, python, pytorch
Domain
data, operating-systems

Research direction

Start by reading fsspec/asyn.py around lines 305-307 and the linked fork-safety guard. Reproduce the failure with an S3-backed filesystem used from a PyTorch DataLoader worker, then determine the supported fork behavior and acceptance criteria. Done should include safe child-process use without reaching into fsspec internals, plus regression coverage for the supported case.

Written by the indexing model from the issue text.

Description

Hi there! Right now, async filesystems are explicitly incompatible with fork, presumably because fsspec tries to make sure that all the asyncio operations actually happen on a separate thread. See https://github.com/fsspec/filesystem_spec/blob/9ea19ba47dcee5c6a0c2435c9a776e8b67d7f7ef/fsspec/asyn.py#L305-L307

I was wondering whether there was some way that we could add limited support for forked processes. I understand that fork + threading gets into some tricky technical weeds, but supporting some level of fork would make for some integration with other projects easier.

In particular, I am trying to use an S3-backed zarr array with PyTorch (in order to train neural networks on some very large arrays) -- torch.utils.data.DataLoader does os.fork by default on Linux (see https://pytorch.org/docs/stable/data.html#single-and-multi-process-data-loading), and the s3fs is an async file system, making this an incompatible combination.

I have been able to hack around this with the following code:

fs = ...   # S3 file system, but could be an other async filesystem
try:
    do_something(fs)
except RuntimeError as err:
    if "fork-safe" not in str(err):
        raise
    import asyncio
    from fsspec import asyn

    if fs._loop is not asyn.loop[0]:
        raise
    # fs._loop is the default loop which is running on a thread on the parent process
    # So let's spin up a new thread for this process.
    new_loop = None
    if len(asyn.loop) == 1:
        new_loop = asyncio.new_event_loop()
        asyn.loop.append(new_loop)  # Appends are thread-safe!
    if asyn.loop[1] is new_loop:   # We inserted first
        th = threading.Thread(target=new_loop.run_forever, name="fsspecIO")
        th.daemon = True
        th.start()
        asyn.iothread.append(th)

    fs._loop = asyn.loop[1]
    fs._pid = os.getpid()

but obviously this is reaching fairly deep into the internals of fsspec, which makes me uncomfortable.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from fsspec/filesystem_spec

All issues in fsspec/filesystem_spec

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.