drivendataorg / drivendataorg/cloudpathlib
Add caching to `download_to` and `upload_from`
- Dominant language
- Python
- Stars
- 628
- Forks
- 88
- Avg merge
- 17h 28m
- Merged PRs (30d)
- 2
Description
Is there any reason why `download_to` and `upload_from` do not support the caching mechanism and, instead, download from the source/cloud directly again and again?
```python
# explicitly instantiate a client that always uses the local cache
LOCAL_CACHE = Path.home() / "cloudpathlib_cache"
client = S3Client(local_cache_dir=LOCAL_CACHE)
local_path = Path("image.png")
ladi = client.CloudPath("s3://ladi/Images/FEMA_CAP/2020/70349")
flood_image = ladi / "DSC_0002_a89f1b79-786f-4dac-9dcc-609fb1a977b1.jpg"
flood_image.download_to(local_path)
LOCAL_CACHE.exists() # False, the cache was not created
flood_image._local.exists() # False, the image was not saved in the cache
# call .fspath to create cached copy
flood_image.fspath
# Now True
LOCAL_CACHE.exists()
flood_image._local.exists()
```
The only disadvantage of the `download_to` method using caching is that you have twice the storage use (the same file is saved in the cache and in `local_path`). But, on the other hand, repeatedly downloading the same data if it hasn't changed also seems rather wasteful to me, especially if it's a large folder. Copying to/from the cache seems much better.
Maybe we can add an argument such as `force_download` or `use_caching` to allow users to enable caching, while keeping the current behaviour the default ( to avoid unexpectedly using twice storage space for those who don't want it).
Alternatively, a cleverer approach that sidesteps the 2x storage use is to make `download_to` and `upload_from` obey the same mechanisms that files in the cache obey when deciding whether to re-download a file or not: check if something already exists in the path given, compared with the cloud version to see if it's outdated, and if not, don't download again.
Finally, what's the use case for this? I don't want to use `open` on a `CloudPath` object directly. For one, it's easier for the user to understand that the file is available locally and they can see it somewhere they define.
Basically, I'm proposing enabling something like the AWSCLI's [`s3 sync` command](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/sync.html) within `cloudpathlib`, since AWS for whatever reason won't add it natively to boto3 itself ([someone asked more than 7 years ago now](https://github.com/boto/boto/issues/3343)). `cloudpathlib` already has the nuts and bolts for this built-in, and it's just a matter of enabling the feature.
Contributor guide
Assessment
This issue has not been assessed yet.