IndexError: list index out of range when 400 HTTP errors

If we get a 400 HTTP error while trying to open a URL with:

fsspec.open(url)

this is not caught and instead an IndexError is raised:

IndexError: list index out of range

See for example:

  File "/src/services/worker/.venv/lib/python3.9/site-packages/fsspec/core.py", line 419, in open
    return open_files(
  File "/src/services/worker/.venv/lib/python3.9/site-packages/fsspec/core.py", line 272, in open_files
    fs, fs_token, paths = get_fs_token_paths(
  File "/src/services/worker/.venv/lib/python3.9/site-packages/fsspec/core.py", line 586, in get_fs_token_paths
    fs = filesystem(protocol, **inkwargs)
  File "/src/services/worker/.venv/lib/python3.9/site-packages/fsspec/registry.py", line 253, in filesystem
    return cls(**storage_options)
  File "/src/services/worker/.venv/lib/python3.9/site-packages/fsspec/spec.py", line 76, in __call__
    obj = super().__call__(*args, **kwargs)
  File "/src/services/worker/.venv/lib/python3.9/site-packages/fsspec/implementations/zip.py", line 54, in __init__
    fo = fsspec.open(
  File "/src/services/worker/.venv/lib/python3.9/site-packages/fsspec/core.py", line 419, in open
    return open_files(
  File "/src/services/worker/.venv/lib/python3.9/site-packages/fsspec/core.py", line 194, in __getitem__
    out = super().__getitem__(item)
IndexError: list index out of range

I have made an investigation with a ZIP archive in Zenodo and with multiprocessing in Python 3.10.11, and there is indeed an underlying 429 HTTP error: Too Many Requests. See:

• huggingface/datasets#5862
• huggingface/datasets#5926

Maybe related to:

• #1256

Start by reproducing the failure with fsspec.open(url) against a URL returning the reported 400 or 429 response. Read fsspec/core.py around open_files and getitem, then inspect fsspec/implementations/zip.py where the nested open call appears; done means the underlying HTTP failure is reported instead of becoming IndexError.