drivendataorg / drivendataorg/cloudpathlib
Add ability to detect if cache is dirty and should be uploaded even if file was not opened by cloudpathlib
- Dominant language
- Python
- Stars
- 628
- Forks
- 88
- Avg merge
- 17h 28m
- Merged PRs (30d)
- 2
Description
We all want to live in a world where every Python library hands PathLike objects. This is not that world.
Many libraries need a path to a local file—especially as a string—in order to read that file. We should expose a supported and documented way to get a path to the version in the cache.
This came up for me in working with `pyvips` where I had to do something like this:
```python
s3p = S3Path("s3://...")
# actually do the download
s3p._refresh_cache()
pyvips.read_image(str(s3p._local))
```
This is not ideal. One option (as noted here: https://github.com/drivendataorg/cloudpathlib/pull/72 ) is to override `__fspath__` to do the caching and return the local path. Then something like this would work:
```python
pyvips.read_image(str(Path(S3Path("s3://...")))
```
Another option (or in addition) is to add a property like `read_only_local_path_string` (🤣 at name)
```python
pyvips.read_image(S3Path("s3://...").read_only_local_path_string)
```
The big caveat is that we replace the `.close` method on the buffer if you open for write through CloudPath so that we know you intend to change the file:
https://github.com/drivendataorg/cloudpathlib/blob/master/cloudpathlib/cloudpath.py#L322-L339
Anything you do to the local path is pretty much read-only since we won't automatically upload. (At least, not without a big change—for example, overriding `__del__` on the CloudPath to do the upload if local is newer, or tracking all of the files DL'd by a Client on the Client and having it check modified times on those and upload. In general I am a little worried about automatically assuming a user wants changed files to be uploaded to overwrite things on the cloud.... )
Contributor guide
Assessment
This issue has not been assessed yet.