dandi / dandi/dandi-cli

Duplicate blob when uploading via CLI

Open
#1,650 3 comments 0 reactions 0 assignees View on GitHub
UX
Dominant language
Python
Stars
28
Forks
37
Avg merge
1d 17h
Merged PRs (30d)
9

Description

I'm running a local instance of dandi-archive (master) using dandi-schema (master).

I've installed dandi cli via `pip install dandi` in a separate venv

```
pip freeze | grep dandi
dandi==0.69.3
dandischema==0.11.1
```

```
$ dandi organize .
$ dandi upload .
```

Output and traceback

```
2025-06-05 13:11:07,679 [ INFO] Found 3 files to consider
PATH SIZE ERRORS PROGRESS STATUS MESSAGE
dandiset.yaml 2.3 kB skipped should be edited online
...-unsorted-nac/sub-Temp-pref-F30-B30-5dpf-01-unsorted-nac_behavior.nwb 9.3 MB pre-validating
...mp-pref-F30-B30-5dpf-01-unsorted-nac_ses-20230808T111800_behavior.nwb 9.3 MB pre-validating
Summary: 18.7 MB 1 skipped 1 should be edited online
2 pre-validating
/home/austin/.pyenv/versions/3.11.12/envs/tmp-dandi-cli/lib/python3.11/site-packages/hdmf/spec/namespace.py:583: UserWarning: Ignoring the following cached namespace(s) because another version is already loaded:
core - cached version: 2.4.0, loaded version: 2.8.0
The loaded extension(s) may not be compatible with the cached extension(s) in the file. Please check the extension documentation and ignore this warning if these versions are compatible.
self.warn_for_ignored_namespaces(ignored_namespaces)
PATH SIZE ERRORS PROGRESS STATUS MESSAGE dandiset.yaml 2.3 kB skipped should be edited online
...-Temp-pref-F30-B30-5dpf-01-unsorted-nac_behavior.nwb 9.3 MB 0 ERROR Error 404 while sending POST reque...
...dpf-01-unsorted-nac_ses-20230808T111800_behavior.nwb 9.3 MB 0 ERROR Error 404 while sending POST reque...
Summary: 18.7 MB 1 skipped 1 should be edited online
2 ERROR 2 Error some while sending POST re...
2025-06-05 13:11:08,841 [ INFO] Logs saved in /home/austin/.local/state/dandi-cli/log/2025.06.05-18.11.07Z-553648.log
Traceback (most recent call last):
File "/home/austin/.pyenv/versions/tmp-dandi-cli/bin/dandi", line 8, in
sys.exit(main())
^^^^^^
File "/home/austin/.pyenv/versions/3.11.12/envs/tmp-dandi-cli/lib/python3.11/site-packages/click/core.py", line 1161, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/austin/.pyenv/versions/3.11.12/envs/tmp-dandi-cli/lib/python3.11/site-packages/click/core.py", line 1082, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/home/austin/.pyenv/versions/3.11.12/envs/tmp-dandi-cli/lib/python3.11/site-packages/click/core.py", line 1697, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/austin/.pyenv/versions/3.11.12/envs/tmp-dandi-cli/lib/python3.11/site-packages/click/core.py", line 1443, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/austin/.pyenv/versions/3.11.12/envs/tmp-dandi-cli/lib/python3.11/site-packages/click/core.py", line 788, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/austin/.pyenv/versions/3.11.12/envs/tmp-dandi-cli/lib/python3.11/site-packages/click/decorators.py", line 45, in new_func
return f(get_current_context().obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/austin/.pyenv/versions/3.11.12/envs/tmp-dandi-cli/lib/python3.11/site-packages/dandi/cli/base.py", line 126, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/austin/.pyenv/versions/3.11.12/envs/tmp-dandi-cli/lib/python3.11/site-packages/dandi/cli/cmd_upload.py", line 103, in upload
upload(
File "/home/austin/.pyenv/versions/3.11.12/envs/tmp-dandi-cli/lib/python3.11/site-packages/dandi/upload.py", line 461, in upload
raise upload_err
File "/home/austin/.pyenv/versions/3.11.12/envs/tmp-dandi-cli/lib/python3.11/site-packages/dandi/upload.py", line 359, in process_path
for r in dfile.iter_upload(
File "/home/austin/.pyenv/versions/3.11.12/envs/tmp-dandi-cli/lib/python3.11/site-packages/dandi/files/bases.py", line 363, in iter_upload
resp = client.post(
^^^^^^^^^^^^
File "/home/austin/.pyenv/versions/3.11.12/envs/tmp-dandi-cli/lib/python3.11/site-packages/dandi/dandiapi.py", line 317, in post
return self.request("POST", path, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/austin/.pyenv/versions/3.11.12/envs/tmp-dandi-cli/lib/python3.11/site-packages/dandi/dandiapi.py", line 285, in request
raise HTTP404Error(msg, response=result)
dandi.exceptions.HTTP404Error: Error 404 while sending POST request to http://localhost:8000/api/uploads/initialize/: {"detail":"Not found."}
```

The upload has made it into my dandiset locally, but we do have that traceback. Looking closer, I think this happens because `dandi organize` created a symlink, and `dandi upload` essentially tries to upload the same file twice.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.