Parquet reference files from git with simplecache

Hi,

I thought it might be a good idea to put the lazy Reference parquet files into git. Using this data directly from git is somehow not possible - e.g. our gitlab server also do not allow byte-range requests which are required at some point, I guess.

So I thought I could add a simplecache:: in the URL and ended up with a catalog which contains entries configured like this:

  pressure-level_analysis_daily:
    args:
      chunks: auto
      consolidated: false
      storage_options:
        lazy: true
        remote_protocol: http
      urlpath: reference::simplecache::{{CATALOG_DIR}}/disk/E5pl00_1D/combined.parq
    driver: zarr

I thought that this triggers downloads of required reference files first before opening it. For opening (getting metadata and coordinates) it seems to work. But for getting the real variable data, it seems like the caching is not fast enough or not syncronized correctly. Especially if I use it with dask, I get a lot of OS errors or incomplete parquet errors when accessing it the first time. When accessing it a third time, it usually works then - as if the caching has only finished then.

Is there a way to configure the process to wait for the caching before using the parquets? Or am I on the wrong track here?

Btw I also tried simplecache::reference:: but then all data that is used and referenced in the reference files is also cached. I only want to cache the reference parquets however...

Thanks and best,
Fabi

Start with the catalog example in main.yaml and reproduce the first and later accesses to the simplecache::reference:: URL, including the dask case. Compare the caching behavior with simplecache::reference:: and determine whether the reference parquet files can be synchronized before use; done means documenting the supported configuration or a confirmed fix for the incomplete-parquet errors.