Allow for a `{key}` to be also present locally as a `{key}.s3invsync/` folder
- Dominant language
- Rust
- Stars
- 3
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
To provide a workaround for rare but possible use-cases
- #170
- #172
and under assumptions that
- this is a rare situation (hence @aaronkanzer [is ok to just skip such problematic paths](https://github.com/dandi/s3invsync/issues/170#issuecomment-2695312031))
and relying on the fact that we do have `.s3invsync.versions.json` where we store version information per "file" in any folder, I think we can workaround by extending record there with optional `bool`, "folder" field, e.g.
```
"{key}": {
"version_id": "...",
"etag": "...",
"folder": true
},
```
which would signify that there is `{key}/` folder on S3. Locally we would keep it as `{key}.s3invsync/` and thus using that for prefix for any path under.
- whenever that `{key}` latest version becomes a `DeleteMarker` (removed), we simply `mv {key}.s3invsync {key}`, and remove that `{key}` record from corresponding `.s3invsync.versions.json`.
- if `{key}` file appears while there is still `{key}/` with some files under, we `mv {key} {key}.s3invsync` first for that folder and add `"folder": true` to the `.s3invsync.versions.json`
Features:
- If `{key}` file (not deleted) is the most recent version (ref: #172) -- we still have access to it just fine under non-modified path, and under `{key}.s3invsync/` we just have older versions (require logic to comprehend anyways)
- If it is a "legit" zarr, there would be no `{key}` file for an existing `{key}/` folder (which would be renamed); and thus we would not have that `{key}.s3invsync/` folder whenever there is an expectation for having `{key}/` directory. So we should be good for that too.
Cons:
- some performance hit since now additional treatments of paths would be needed but I think it could be quite minimal since once again
WDYT @jwodder ?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.