dandi / dandi/dandi-cli

Support partial zarr upload

Open
#1,462 19 comments 0 reactions 0 assignees View on GitHub
cmd-upload UX zarr
Dominant language
Python
Stars
28
Forks
37
Avg merge
1d 17h
Merged PRs (30d)
9

Description

Original use-case

- https://github.com/dandi/dandi-cli/issues/1474

to modify some metadata files within zarr. Zarr itself might be huge (GBs) so it is a big waste to require download of the entire zarr folder to just update some metadata files. So relates also to ability for partial download

- dandi/dandi-archive#1461

`dandi upload` if operating on a `.zarr` folder, should gain an option so upload only considers paths present but does not assume that it is "all of zarr" and remove the rest from remote zarr. Then it needs to finalize zarr update with full manifest as with some files/folders of zarr present only in archive, and not locally.

I think it should be `--zarr-mode full,partial-full,partial-lean`. Modes
- `full` -- current behavior (pretty much a sync IIRC)
- `dont-delete` -- upload/update files found locally but do not delete any remote file within zarr (added later)

Previously thought about (edited):
- `partial-full` -- if encountering zarr sub-folder locally, it would pretty much do `full` on that subfolder -- i.e. it would remove remote files under that subfolder if not present locally. As a "side-effect" -- if encounters an empty sub-folder locally -- it would remove all the zarr subfiles potentially present with that folder prefix on archive (thus it would be easy to prune some undesired remote folders).
- `partial-lean` -- it would only concern with updates of files it encounters.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.