dandi / dandi/dandi-cli

Upload is inconsistent with NWB Zarr files

Open
#1,520 11 comments 0 reactions 0 assignees View on GitHub
zarr
Dominant language
Python
Stars
28
Forks
37
Avg merge
1d 17h
Merged PRs (30d)
9

Description

I am able to upload a small zarr file to DANDI (~56Mb). However, I am trying to upload a file (~34Gb) and at first I am able to reach 92% upload after a span of 36 hrs. I got the error shown. I assume it was a time out issue so I ran it again. I was told that if I ran the same command I did prior (dandi upload --existing-refresh) that it should continue from the last checkpoint.

I got the same error the second time, but now at 32%

Questions:
1. Is that 32% of what is left over or did it start up again?
2. Why is this happening and would decreasing the number of files (increasing the chunk size) in the zarr store help (currently at 1.5 million)

Screenshot 2024-11-05 at 6 05 48 PM

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.