"File name too long" with HTTP presigned requests

Discussed in https://github.com/fsspec/filesystem_spec/discussions/1490

^{Originally posted by cgrass January 4, 2024}
Hello,
I'm new to python and fsspec, so hopefully there is an obvious answer to my question. Thanks for the help!

Issue

I need to fetch a file from an s3 bucket using https and a pre-signed URL. The URL produced is quite long, commonly over 1500 characters (mostly query params). If I use the code below, I get an OS exception: File name too long.

def test_this():
    kwargs = {}
    spec = fsspec.filesystem("https", **kwargs)
    frompath = {longurl}
    topath = "/var/"  ### in prod this is a real dest
    response = spec.get(frompath, topath, recursive=False, **kwargs)
    print(response)

I walked through the code and can see that the rpath is typically appended to the lpath for storage in asyn.py:

lpaths = other_paths(
    rpaths,
    lpath,
    exists=exists,
    flatten=not source_is_str,
)

def other_paths(
    paths: list[str],
    path2: str | list[str],
    exists: bool = False,
    flatten: bool = False,
) -> list[str]:
    if isinstance(path2, str):
        path2 = path2.rstrip("/")

        if flatten:
            path2 = ["/".join((path2, p.split("/")[-1])) for p in paths]
        else:
            cp = common_prefix(paths)
            if exists:
                cp = cp.rsplit("/", 1)[0]
            if not cp and all(not s.startswith("/") for s in paths):
                path2 = ["/".join([path2, p]) for p in paths]
            else:
                path2 = [p.replace(cp, path2, 1) for p in paths]
    else:
        assert len(paths) == len(path2)
    return path2

Question

Am I misusing the lib or protocol? Is there a way to configure fsspec to scrub query params (or use some arbitrary string) for the appended destination string?

Setup

MacOS Ventura M1 Pro
fsspec-2023.12.2
Python 3.9, 3.10 (tried both)

Linked bug report

Continue with the linked bug report in issue 1505, then inspect the path handling in asyn.py, especially other_paths(). Reproduce the failure with a long presigned HTTPS URL and verify that destination path handling no longer produces an invalid local filename, with regression coverage for the reported case.