upload+download: RF to reuse logic/options/UI
- Dominant language
- Python
- Stars
- 28
- Forks
- 37
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 9
Description
ATM implementations are separate and analysis for either to download/upload a file is happening right before actual download/upload. That forbids implementing proper "sync" functionality where analysis first to be done on either any file needs to be removed (or may be moved! ;-) ) on local/server end. So RF should be done to minimize difference between API and implementation of the two:
- given the path specification both should first obtain list of local/remote "assets"
- given the mode of operation in case of "existing" do the analysis and provide user with the summary on upcoming transfer
- possibly even present user with a list of files which would not be download/uploaded because they are already newer on the destination
- pass into actual download/upload functions only the files which decided to be acted upon
- if it was full dandiset (or a folder) to be downloaded/uploaded -- perform necessary deletions (locally or remotely) (might want to be a dedicated option, e.g. `--sync`)
- exit with non-0 if any file which could have been download/uploaded was not (e.g. in case of `--sync` and `--file-mode` not being "overwrite" or "force", if some file was already newer and we didn't perform transfer)
For download would be needed:
- to not use girder's downloadFile, which relies on a context manager to report progress, but which gets only filename as an ID.
- Also it downloads into temporary directory which might be on a different partition. Should be near the target file. we could then detect/continue interrupted downloads etc
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.