dandi / dandi/s3invsync

Allow for a `{key}` to be also present locally as a `{key}.s3invsync/` folder

Open
#174 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
3
Forks
1
PR merge metrics
No merged PRs in 30d

Description

To provide a workaround for rare but possible use-cases

- #170
- #172

and under assumptions that

- this is a rare situation (hence @aaronkanzer [is ok to just skip such problematic paths](https://github.com/dandi/s3invsync/issues/170#issuecomment-2695312031))

and relying on the fact that we do have `.s3invsync.versions.json` where we store version information per "file" in any folder, I think we can workaround by extending record there with optional `bool`, "folder" field, e.g.

```
"{key}": {
"version_id": "...",
"etag": "...",
"folder": true
},
```

which would signify that there is `{key}/` folder on S3. Locally we would keep it as `{key}.s3invsync/` and thus using that for prefix for any path under.

- whenever that `{key}` latest version becomes a `DeleteMarker` (removed), we simply `mv {key}.s3invsync {key}`, and remove that `{key}` record from corresponding `.s3invsync.versions.json`.
- if `{key}` file appears while there is still `{key}/` with some files under, we `mv {key} {key}.s3invsync` first for that folder and add `"folder": true` to the `.s3invsync.versions.json`

Features:
- If `{key}` file (not deleted) is the most recent version (ref: #172) -- we still have access to it just fine under non-modified path, and under `{key}.s3invsync/` we just have older versions (require logic to comprehend anyways)
- If it is a "legit" zarr, there would be no `{key}` file for an existing `{key}/` folder (which would be renamed); and thus we would not have that `{key}.s3invsync/` folder whenever there is an expectation for having `{key}/` directory. So we should be good for that too.

Cons:
- some performance hit since now additional treatments of paths would be needed but I think it could be quite minimal since once again

WDYT @jwodder ?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.