drivendataorg / drivendataorg/cloudpathlib

Cache timestamps are wrong while downloading

Open
#522 2 comments 0 reactions 0 assignees View on GitHub
caching
Dominant language
Python
Stars
628
Forks
88
Avg merge
17h 28m
Merged PRs (30d)
2

Description

When cloudpathliub sees a file for the first time it start downloading it to the cache location and only once the download is complete it sets its timestamps to the date read from the cloud version. This means that while the file is being downloaded it has the wrong (current date) timestamps!

This is normally not an issue because cloudpathlib uses a tmpdir but it also allows setting `CLOUDPATHLIB_LOCAL_CACHE_DIR` and this would cause issues if you spin up multiple processes that access the same file:

For example if you're dealing with a large file downloading it could take a while and the cached file has the wrong date in that period. If you start a second copy of your program before the download finishes then the second instance sees the incomplete cached file with the wrong date and raises `cloudpathlib.exceptions.OverwriteNewerLocalError`.

Even if the file had the correct date as it was being downloaded by the first process the second process wouldn't know if the download has finished or not.

I think at least the documentation for `CLOUDPATHLIB_LOCAL_CACHE_DIR` should be updated to mention that its not multi thread/process safe.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.