dandi / dandi/dandi-cli

upload+download: RF to reuse logic/options/UI

Open
#48 6 comments 0 reactions 0 assignees View on GitHub
cmd-download cmd-upload
Dominant language
Python
Stars
28
Forks
37
Avg merge
1d 17h
Merged PRs (30d)
9

Description

ATM implementations are separate and analysis for either to download/upload a file is happening right before actual download/upload. That forbids implementing proper "sync" functionality where analysis first to be done on either any file needs to be removed (or may be moved! ;-) ) on local/server end. So RF should be done to minimize difference between API and implementation of the two:

- given the path specification both should first obtain list of local/remote "assets"
- given the mode of operation in case of "existing" do the analysis and provide user with the summary on upcoming transfer
- possibly even present user with a list of files which would not be download/uploaded because they are already newer on the destination
- pass into actual download/upload functions only the files which decided to be acted upon
- if it was full dandiset (or a folder) to be downloaded/uploaded -- perform necessary deletions (locally or remotely) (might want to be a dedicated option, e.g. `--sync`)
- exit with non-0 if any file which could have been download/uploaded was not (e.g. in case of `--sync` and `--file-mode` not being "overwrite" or "force", if some file was already newer and we didn't perform transfer)

For download would be needed:

- to not use girder's downloadFile, which relies on a context manager to report progress, but which gets only filename as an ID.
- Also it downloads into temporary directory which might be on a different partition. Should be near the target file. we could then detect/continue interrupted downloads etc

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.