Support partial zarr upload
- Dominant language
- Python
- Stars
- 28
- Forks
- 37
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 9
Description
Original use-case
- https://github.com/dandi/dandi-cli/issues/1474
to modify some metadata files within zarr. Zarr itself might be huge (GBs) so it is a big waste to require download of the entire zarr folder to just update some metadata files. So relates also to ability for partial download
- dandi/dandi-archive#1461
`dandi upload` if operating on a `.zarr` folder, should gain an option so upload only considers paths present but does not assume that it is "all of zarr" and remove the rest from remote zarr. Then it needs to finalize zarr update with full manifest as with some files/folders of zarr present only in archive, and not locally.
I think it should be `--zarr-mode full,partial-full,partial-lean`. Modes
- `full` -- current behavior (pretty much a sync IIRC)
- `dont-delete` -- upload/update files found locally but do not delete any remote file within zarr (added later)
Previously thought about (edited):
- `partial-full` -- if encountering zarr sub-folder locally, it would pretty much do `full` on that subfolder -- i.e. it would remove remote files under that subfolder if not present locally. As a "side-effect" -- if encounters an empty sub-folder locally -- it would remove all the zarr subfiles potentially present with that folder prefix on archive (thus it would be easy to prune some undesired remote folders).
- `partial-lean` -- it would only concern with updates of files it encounters.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.