concurrent.futures.ProcessPoolExecutor fails for HTTPFileSystem

@CodyCBakerPhD and I have been exploring different ways of parallelizing with http fsspec. We have found that you can use multiple processes in joblib (1), and parallelizing over threads using concurrent.futures.ThreadPoolExecutor works (2), but concurrent.futures.ProcessPoolExecutor stalls.

1. Parallel, asynchronous access with joblib

this works:

from joblib import Parallel, delayed

def get_subject_ids_and_session_start_times(s3_url: str) -> tuple:
    file_system_per_run = fsspec.filesystem("http")

    with (
        file_system_per_run.open(path=s3_url, mode="rb") as name,
        h5py.File(name=name) as file,
    ):
        return str(file.keys())

results = Parallel(n_jobs=3)(delayed(get_subject_ids_and_session_start_times)(s3_url) for s3_url in s3_urls)
for output in results:
    print(output)

2. Parallel, asynchronous access with concurrent.futures.ThreadPoolExecutor

this works:

import concurrent.futures

def get_subject_ids_and_session_start_times(s3_url: str) -> tuple:
    file_system_per_run = fsspec.filesystem("http")

    with (
        file_system_per_run.open(path=s3_url, mode="rb") as name,
        h5py.File(name=name) as file,
    ):
        return str(file.keys())

with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
    futures = [executor.submit(get_subject_ids_and_session_start_times, s3_url) for s3_url in s3_urls]
for future in concurrent.futures.as_completed(futures):
    print(future.result())

3. Parallel, asynchronous access with concurrent.futures.ProcessPoolExecutor

this stalls:

import concurrent.futures

def get_subject_ids_and_session_start_times(s3_url: str) -> tuple:
    file_system_per_run = fsspec.filesystem("http")

    with (    
        file_system_per_run.open(path=s3_url, mode="rb") as name,
        h5py.File(name=name) as file,
    ):
        return str(file.keys())

with concurrent.futures.ProcessPoolExecutor(max_workers=3) as executor:
    futures = [executor.submit(get_subject_ids_and_session_start_times, s3_url) for s3_url in s3_urls]
for future in concurrent.futures.as_completed(futures):
    print(future.result())

@martindurant , any idea why this third method might fail where the other two succeed? Is this third approach in scope for fsspec?

Start by reproducing the HTTPFileSystem example with ProcessPoolExecutor, then compare it with the working ThreadPoolExecutor and joblib examples. Inspect the HTTPFileSystem process-pool interaction and determine whether the stall is a defect or an unsupported usage; done means the cause and supported behavior are established, with a regression test or documentation change if appropriate.