drivendataorg / drivendataorg/cloudpathlib

Implement caching based on file sizes and file content where possible

Open
#12 1 comment 0 reactions 0 assignees View on GitHub
caching enhancement
Dominant language
Python
Stars
628
Forks
88
Avg merge
17h 28m
Merged PRs (30d)
2

Description

Sometimes we can use etags, Azure provides the md5 hash, some providers have "versions," and there may be other options. Currently trusting file times is a little flaky.

Can we store the etag info on the local file somehow natively? It would be good if this metadata was around to be used across processes, etc.

We aren't doing this now, but we should check file sizes before we even check times. If sizes are different, the files are definitely different. Then we can fallback to if times are different, they are possibly different. Then, hashes can confirm files are different.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.