Duplicate blob when uploading via CLI
- Dominant language
- Python
- Stars
- 28
- Forks
- 37
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 9
Description
I'm running a local instance of dandi-archive (master) using dandi-schema (master).
I've installed dandi cli via `pip install dandi` in a separate venv
```
pip freeze | grep dandi
dandi==0.69.3
dandischema==0.11.1
```
```
$ dandi organize .
$ dandi upload .
```
Output and traceback
```
2025-06-05 13:11:07,679 [ INFO] Found 3 files to consider
PATH SIZE ERRORS PROGRESS STATUS MESSAGE
dandiset.yaml 2.3 kB skipped should be edited online
...-unsorted-nac/sub-Temp-pref-F30-B30-5dpf-01-unsorted-nac_behavior.nwb 9.3 MB pre-validating
...mp-pref-F30-B30-5dpf-01-unsorted-nac_ses-20230808T111800_behavior.nwb 9.3 MB pre-validating
Summary: 18.7 MB 1 skipped 1 should be edited online
2 pre-validating
/home/austin/.pyenv/versions/3.11.12/envs/tmp-dandi-cli/lib/python3.11/site-packages/hdmf/spec/namespace.py:583: UserWarning: Ignoring the following cached namespace(s) because another version is already loaded:
core - cached version: 2.4.0, loaded version: 2.8.0
The loaded extension(s) may not be compatible with the cached extension(s) in the file. Please check the extension documentation and ignore this warning if these versions are compatible.
self.warn_for_ignored_namespaces(ignored_namespaces)
PATH SIZE ERRORS PROGRESS STATUS MESSAGE dandiset.yaml 2.3 kB skipped should be edited online
...-Temp-pref-F30-B30-5dpf-01-unsorted-nac_behavior.nwb 9.3 MB 0 ERROR Error 404 while sending POST reque...
...dpf-01-unsorted-nac_ses-20230808T111800_behavior.nwb 9.3 MB 0 ERROR Error 404 while sending POST reque...
Summary: 18.7 MB 1 skipped 1 should be edited online
2 ERROR 2 Error some while sending POST re...
2025-06-05 13:11:08,841 [ INFO] Logs saved in /home/austin/.local/state/dandi-cli/log/2025.06.05-18.11.07Z-553648.log
Traceback (most recent call last):
File "/home/austin/.pyenv/versions/tmp-dandi-cli/bin/dandi", line 8, in
sys.exit(main())
^^^^^^
File "/home/austin/.pyenv/versions/3.11.12/envs/tmp-dandi-cli/lib/python3.11/site-packages/click/core.py", line 1161, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/austin/.pyenv/versions/3.11.12/envs/tmp-dandi-cli/lib/python3.11/site-packages/click/core.py", line 1082, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/home/austin/.pyenv/versions/3.11.12/envs/tmp-dandi-cli/lib/python3.11/site-packages/click/core.py", line 1697, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/austin/.pyenv/versions/3.11.12/envs/tmp-dandi-cli/lib/python3.11/site-packages/click/core.py", line 1443, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/austin/.pyenv/versions/3.11.12/envs/tmp-dandi-cli/lib/python3.11/site-packages/click/core.py", line 788, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/austin/.pyenv/versions/3.11.12/envs/tmp-dandi-cli/lib/python3.11/site-packages/click/decorators.py", line 45, in new_func
return f(get_current_context().obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/austin/.pyenv/versions/3.11.12/envs/tmp-dandi-cli/lib/python3.11/site-packages/dandi/cli/base.py", line 126, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/austin/.pyenv/versions/3.11.12/envs/tmp-dandi-cli/lib/python3.11/site-packages/dandi/cli/cmd_upload.py", line 103, in upload
upload(
File "/home/austin/.pyenv/versions/3.11.12/envs/tmp-dandi-cli/lib/python3.11/site-packages/dandi/upload.py", line 461, in upload
raise upload_err
File "/home/austin/.pyenv/versions/3.11.12/envs/tmp-dandi-cli/lib/python3.11/site-packages/dandi/upload.py", line 359, in process_path
for r in dfile.iter_upload(
File "/home/austin/.pyenv/versions/3.11.12/envs/tmp-dandi-cli/lib/python3.11/site-packages/dandi/files/bases.py", line 363, in iter_upload
resp = client.post(
^^^^^^^^^^^^
File "/home/austin/.pyenv/versions/3.11.12/envs/tmp-dandi-cli/lib/python3.11/site-packages/dandi/dandiapi.py", line 317, in post
return self.request("POST", path, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/austin/.pyenv/versions/3.11.12/envs/tmp-dandi-cli/lib/python3.11/site-packages/dandi/dandiapi.py", line 285, in request
raise HTTP404Error(msg, response=result)
dandi.exceptions.HTTP404Error: Error 404 while sending POST request to http://localhost:8000/api/uploads/initialize/: {"detail":"Not found."}
```
The upload has made it into my dandiset locally, but we do have that traceback. Looking closer, I think this happens because `dandi organize` created a symlink, and `dandi upload` essentially tries to upload the same file twice.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.