drivendataorg / drivendataorg/cloudpathlib

Add ability to detect if cache is dirty and should be uploaded even if file was not opened by cloudpathlib

Open
#73 5 comments 0 reactions 0 assignees View on GitHub
caching enhancement
Dominant language
Python
Stars
628
Forks
88
Avg merge
17h 28m
Merged PRs (30d)
2

Description

We all want to live in a world where every Python library hands PathLike objects. This is not that world.

Many libraries need a path to a local file—especially as a string—in order to read that file. We should expose a supported and documented way to get a path to the version in the cache.

This came up for me in working with `pyvips` where I had to do something like this:
```python
s3p = S3Path("s3://...")

# actually do the download
s3p._refresh_cache()

pyvips.read_image(str(s3p._local))
```

This is not ideal. One option (as noted here: https://github.com/drivendataorg/cloudpathlib/pull/72 ) is to override `__fspath__` to do the caching and return the local path. Then something like this would work:

```python
pyvips.read_image(str(Path(S3Path("s3://...")))
```

Another option (or in addition) is to add a property like `read_only_local_path_string` (🤣 at name)

```python
pyvips.read_image(S3Path("s3://...").read_only_local_path_string)
```

The big caveat is that we replace the `.close` method on the buffer if you open for write through CloudPath so that we know you intend to change the file:
https://github.com/drivendataorg/cloudpathlib/blob/master/cloudpathlib/cloudpath.py#L322-L339

Anything you do to the local path is pretty much read-only since we won't automatically upload. (At least, not without a big change—for example, overriding `__del__` on the CloudPath to do the upload if local is newer, or tracking all of the files DL'd by a Client on the Client and having it check modified times on those and upload. In general I am a little worried about automatically assuming a user wants changed files to be uploaded to overwrite things on the cloud.... )

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.