dandi / dandi/dandi-cli

necessary steps for syncing out-of-band upload of dandiset 108

Open
#956 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
28
Forks
37
Avg merge
1d 17h
Merged PRs (30d)
9

Description

for dandiset 108, we have been doing out of band uploads and recording locally the connection between asset id and zarr id. this was done because the asset ids disappeared for a batch of uploads at one point in the staging server.

at present:
- dandiset 108 contains both old assets and new assets. these are organized by sessions within folders.
- however, these new assets do not have appropriate metadata
- there are some assets that have not been uploaded. i would like to use the CLI to test upload performance of some of these.

questions:
1. what specific CLI steps need to be followed to sync the assets and dandisets?
2. can asset complete be forced through a debug flag if directly not available? or can the CLI force if it sees metadata does not match file count/size or is missing asset checksum?
3. how should i quantify operation times? are there debug flags that can be enabled to record time to check? (where operations include: are the files all available remotely, upload time, )
4. i am assuming that CLI upload is still slower than out of band upload + CLI check. has this been tested. if not can someone test this?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.