DirFileSystem.ls(path, detail=True) hits assertion error with Azure Blob Storage

I ran into an error using DirFileSystem with Azure Blob Storage. Here's the minimum code to expose this error:

import fsspec
from fsspec.implementations.dirfs import DirFileSystem

bucket = "my-bucket"

fs = fsspec.filesystem("abfs")
dirfs = DirFileSystem(path=bucket, fs=fs)
with fs.open(f"{bucket}/some-dir/some-file.txt", "w") as f:
    f.write("Hello, fsspec!")

print(dirfs.ls("some-dir", detail=True))
print(dirfs.ls("some-dir", detail=True))

When I run it, the first dirfs.ls() call works fine, but the second hits this AssertionError:

Traceback (most recent call last):
  File "test_dirfs_min.py", line 12, in <module>
    print(dirfs.ls("some-dir", detail=True))
  File "/usr/local/lib/python3.8/site-packages/fsspec/implementations/dirfs.py", line 213, in ls
    entry["name"] = self._relpath(entry["name"])
  File "/usr/local/lib/python3.8/site-packages/fsspec/implementations/dirfs.py", line 41, in _relpath
    assert path.startswith(prefix)
AssertionError

I believe this error happens because the first time dirfs.ls() is called, when the entry["name"] = self._relpath(entry["name"]) line runs, that modifies the object(s?) that are cached. Then the second time dirfs.ls() is called, the accidentally-modified cache is returned in this line (looks like this in case that line changes in the future)

ret = self.fs.ls(self._join(path), detail=detail, **kwargs)

Then the paths returned don't start with the prefix, triggering the AssertionError above.

I haven't traced through the caching logic, so there's probably a better way to solve this, but deepcopying the ret object before modifying it fixed the issue for me. See this commit in my fork.

Here's some slightly more extensive code that only fails at the very end with the current master branch, but runs through with my patch. Note that when testing "file", "memory", and "abfs" filesystems, we only run into the issue when using a DirFileSystem with Azure Blob Storage and setting detail=True. All other combinations work fine with the current code.

from pathlib import Path

import fsspec
from fsspec.implementations.dirfs import DirFileSystem

bucket = "my-bucket"

options_dicts = [
    {"fs_type": "file", "base_dir": f"{Path(__file__).parent}/{bucket}"},
    {"fs_type": "memory", "base_dir": bucket},
    {"fs_type": "abfs", "base_dir": bucket},
]

for options in options_dicts:
    fs_type = options["fs_type"]
    base_dir = options["base_dir"]
    print(f"Testing {fs_type}:")
    fs = fsspec.filesystem(fs_type, auto_mkdir=True)
    with fs.open(f"{base_dir}/some-dir/some-file.txt", "w") as f:
        f.write("Hello, fsspec!")
    print(fs.ls(f"{base_dir}/some-dir", detail=False))
    print(fs.ls(f"{base_dir}/some-dir", detail=False))
    print(fs.ls(f"{base_dir}/some-dir", detail=True))
    print(fs.ls(f"{base_dir}/some-dir", detail=True))

    dirfs = DirFileSystem(path=base_dir, fs=fs)
    print(dirfs.ls("some-dir", detail=False))
    print(dirfs.ls("some-dir", detail=False))
    print(dirfs.ls("some-dir", detail=True))
    print(dirfs.ls("some-dir", detail=True))

Would you like me to make a PR from my fork or is there a different way we should go about solving this problem?

Start in fsspec/implementations/dirfs.py, especially ls around line 210 and _relpath around line 41, then reproduce the repeated detail=True call with Azure Blob Storage from the issue. Verify that repeated listings no longer trigger AssertionError and that the broader file, memory, and abfs examples retain their expected results.