Refresh local cache when accessed if remote file has been updated

From reading the documentation on file-level caching, my understanding is that the only option for expiring the local cache is setting the expiry_time to some fixed value.

However, at least some filesystem backends seem to be able to provide information about the last time a file was modified (e.g. the updated attribute in GCS objects), and it would be convenient if there were a way to only update the local cache when accessing a file if the remote file has been updated since the cache was created.

Is this something that's already possible? I apologize if this is addressed somewhere already, but I searched and didn't find it.

Our use case: We want to publish intake data catalogs for both tagged versions of the outputs from a data pipeline, as well as nightly builds that correspond to various development branches in the pipeline repository. E.g. the outputs from the commit tagged v1.2.3 would be available at gs://intake.catalyst.coop/v1.2.3 while the most recent successful build outputs from the dev branch would be available at gs://intake.catalyst.coop/dev. The tagged version would be static, but the branch reference could change on a daily basis, and it would be good to be able to refresh the local cache automatically if it's updated since the last time it was cached.

I guess maybe this could be done with object versioning too, but I haven't really looked into how that works.

See this issue.

Start with the SimpleCacheFileSystem file-level caching documentation and its expiry_time behavior. Investigate whether filesystem backends expose remote modification timestamps, such as the GCS object updated attribute, and how the cache can compare that value with its local entry. Done means an accessed cached file refreshes only when the remote file has changed, while fixed expiry behavior remains understandable.