`filecache` and `simplecache` both ignore `version_id` for versioned S3 filesystems

Caching S3 data with e.g., simplecache works fine. However, S3FS also supports versioned buckets.

When a version_id is provided to open, using a cache leads to incorrect behavior, as the cache seems to drop the version argument and always requests, and subsequently caches, the latest version.

The code below should reproduce the issue, as long as you provide it with the bucket name for a versioned bucket. Removing the caching fixes the issue.

Possible fixes:

• make caching aware of S3 version IDs (may need to "hack" the local disk cache paths to include the version IDs)
• make caches raise an error when they detect an attempt to wrap a versioned bucket, in order to protect the user from this surprising behavior

Please let me know if I can provide any additional information or code!

import random
import time

import fsspec
import numpy as np
import s3fs


def eval_version_caching():
    bucket = "MY_TEST_BUCKET"

    fs = s3fs.S3FileSystem(version_aware=True)
    print(fs.ls(bucket))

    version_dir = f"tmp/dbg-version-eval-{random.randint(0, 100000):010d}"
    data_path = f"{bucket}/{version_dir}/data.npy"

    # 8 * 2 * 2 Mb, roughly = 32 Mb
    some_data = np.ones((2000, 2000), dtype=np.float64) * 1.0
    some_other_data = np.ones((2000, 2000), dtype=np.float64) * 42.0

    with fs.open(data_path, "wb") as f_out:
        np.save(f_out, some_data)

    print("Saved first version... Waiting to flush.")
    time.sleep(1.0)

    with fs.open(data_path, "wb") as f_out:
        np.save(f_out, some_other_data)

    print("Saved second version... Waiting to flush.")
    time.sleep(1.0)

    versions = fs.object_version_info(data_path)
    assert 2 == len(versions)

    # Versions are listed in reverse chronological order. In our case, we call the first version "a", and the second
    # version "b".
    id_rev_a = versions[1]["VersionId"]
    id_rev_b = versions[0]["VersionId"]
    print(id_rev_a)
    print(id_rev_b)

    # The cached FS always returns the latest version, no matter what version_id we provide.
    cached_fs = fsspec.filesystem(
        "filecache", target_protocol="s3", target_options={"version_aware": True}, cache_storage="/tmp/s3_file_cache"
    )
    # Using this non cached filesystem correctly retrieves the correct versions
    # non_cached_fs = s3fs.S3FileSystem(version_aware=True)
    with cached_fs.open(data_path, "rb", version_id=id_rev_a) as f_out:
        read_data = np.load(f_out)
        print("Revision A:")
        print(read_data.mean())
        rev_a_mean = read_data.mean()

    with cached_fs.open(data_path, "rb", version_id=id_rev_b) as f_out:
        read_data = np.load(f_out)
        print("Revision B:")
        print(read_data.mean())
        rev_b_mean = read_data.mean()

    assert abs(rev_a_mean - rev_b_mean) > 1e-5



def main():
    eval_version_caching()


if __name__ == "__main__":
    main()

Start by reproducing the example with the filecache and simplecache implementations, focusing on how cached_fs.open handles version_id for the versioned S3 filesystem. The work is done when separate version IDs return their corresponding data, or the cache explicitly rejects versioned access; the issue does not name specific source files or tests.