Deadlock involving pandas.read_excel and AWS S3

This may be related to https://github.com/fsspec/filesystem_spec/issues/517, but the workaround suggested there does not seem to work...

# From https://github.com/fsspec/filesystem_spec/issues/517
# Adapted for s3fs

import multiprocessing
import time

import pandas as pd
import s3fs

use_multiprocessing = True


def read_file(path):
    t0 = time.time()

    def elapsed():
        return time.time() - t0

    print(f'{elapsed():.3f} Before read_excel')
    fs = s3fs.S3FileSystem()

    # Workaround?
    fs.clear_instance_cache()

    with fs.open(path, "rb") as f:
        print(f"{elapsed():.3f} Entered context manager for {path}")
        df = pd.read_excel(f)
    print(f'{elapsed():.3f} After read excel')
    return df


try:
    df = pd.read_csv('s3://<nonexistent CSV file>')
except FileNotFoundError:
    pass

filepath = 's3://<existing Excel file>'

files = [filepath]

if use_multiprocessing:
    with multiprocessing.Pool(1) as pool:
        dfs = pool.map(read_file, files)
else:
    dfs = [read_file(p) for p in files]

Running with use_multiprocessing = True results in the program to print out 0.000 Before read_excel and then hanging. With use_multiprocessing = False, the file is read successfully in about 1 second.

I appreciate that asyncio may be difficult to do in multiprocessing environment, but in this case, I am rather stymied by the fact that there should be no shared state anywhere between the processes, as a completely fresh S3FileSystem instance is created in each subprocess. The suggested workaround of invoking clear_instance_cache() did not work either.

In my specific case, I ended up removing the s3fs dependency completely and use boto3 to download the object to a BytesIO, which is then happily ingested by Pandas (the files I have in this project are rather small, less than 1 MB). However, since Pandas uses s3fs and thus fsspec under the hood, I would like to confirm that this issue indeed persists and if so, ask Pandas maintainers to add a note to their documentation about the issue.

fsspec version 2021.11.1, Python 3.8.10, pandas 1.3.5

Start by running the provided multiprocessing.Pool reproducer with the stated Python, fsspec, and pandas versions, comparing it with the sequential path. Read the interaction between S3FileSystem construction, clear_instance_cache(), and pandas.read_excel. Done means identifying whether the hang persists and documenting or addressing the confirmed behavior in the appropriate project guidance.