Add check/handle gracefully attempt to upload/rename asset to have path longer than 512 characters
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 26
- Forks
- 21
- Avg merge
- 4d 23h
- Merged PRs (30d)
- 15
Description
512 is current length limit for the path, and it is more than enough. (the longest path in archive I found is 192:
dandi@drogon:/mnt/backup/dandi/dandisets$ declare m=0;for ds in 00*; do git -C $ds ls-tree -r --name-only HEAD; done | awk '{ if (length($0) > max) max = length($0) } END { print max }'
192
dandi@drogon:/mnt/backup/dandi/dandisets$ pwd
/mnt/backup/dandi/dandisets
but if I do attempt to upload a file with path length greater -- it causes 500
❯ DANDI_DEVEL=1 dandi upload --allow-any-path $fn
2024-03-06 20:34:17,039 [ INFO] Found 2 files to consider
PATH SIZE ERRORS PROGRESS STATUS MESSAGE
dandiset.yaml 3.0 kB skipped should be ed...
...901234567890123456789_data.dat 4 Bytes 0 producing asset
Summary: 3.1 kB 1 skipped 1 should be ...
1 producing ...
...901234567890123456789_data.dat 4 Bytes 0 ERROR 500 Server E...
Summary: 3.1 kB 1 skipped 1 should be ...
1 ERROR 1 500 Server...
2024-03-06 20:35:13,412 [ INFO] Logs saved in /home/yoh/.local/state/dandi-cli/log/20240307013414Z-313889.log
Traceback (most recent call last):
File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/bin/dandi", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/click/decorators.py", line 45, in new_func
return f(get_current_context().obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yoh/proj/dandi/dandi-cli-master/dandi/cli/base.py", line 126, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/yoh/proj/dandi/dandi-cli-master/dandi/cli/cmd_upload.py", line 103, in upload
upload(
File "/home/yoh/proj/dandi/dandi-cli-master/dandi/upload.py", line 459, in upload
raise upload_err
File "/home/yoh/proj/dandi/dandi-cli-master/dandi/upload.py", line 357, in process_path
for r in dfile.iter_upload(
File "/home/yoh/proj/dandi/dandi-cli-master/dandi/files/bases.py", line 455, in iter_upload
r = client.post(
^^^^^^^^^^^^
File "/home/yoh/proj/dandi/dandi-cli-master/dandi/dandiapi.py", line 306, in post
return self.request("POST", path, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yoh/proj/dandi/dandi-cli-master/dandi/dandiapi.py", line 200, in request
for i, attempt in enumerate(
File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/tenacity/__init__.py", line 347, in __iter__
do = self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/tenacity/__init__.py", line 325, in iter
raise retry_exc.reraise()
^^^^^^^^^^^^^^^^^^^
File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/tenacity/__init__.py", line 158, in reraise
raise self.last_attempt.result()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/home/yoh/proj/dandi/dandi-cli-master/dandi/dandiapi.py", line 236, in request
result.raise_for_status()
File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.dandiarchive.org/api/dandisets/000029/versions/draft/assets/
DANDI_DEVEL=1 dandi upload --allow-any-path $fn 2.27s user 2.93s system 8% cpu 58.988 total
❯
❯ echo $fn
sub-123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789/sub-123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789_ses-1/sub-123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789_data.dat
❯ echo ${#fn}
596
and as we on 500 re-try -- poor dandi-archive keeps 500ing:
2024-03-07T01:40:14.785006+00:00 app[web.1]: 10.1.61.154 - - [07/Mar/2024:01:40:14 +0000] "POST /api/dandisets/000029/versions/draft/assets/ HTTP/1.1" 500 145 "-" "dandi/0.60.0+11.gee18a15e requests/2.31.0 CPython/3.11.8"
2024-03-07T01:40:14.785346+00:00 heroku[router]: at=info method=POST path="/api/dandisets/000029/versions/draft/assets/" host=api.dandiarchive.org request_id=4a509b4c-8716-4350-973c-33deaa1761f1 fwd="73.218.136.210" dyno=web.1 connect=0ms service=48ms status=500 bytes=512 protocol=https
2024-03-07T01:40:16.133195+00:00 heroku[router]: at=info method=POST path="/api/dandisets/000029/versions/draft/assets/" host=api.dandiarchive.org request_id=812d665f-2909-42de-88f3-8c400639f8fc fwd="73.218.136.210" dyno=web.1 connect=0ms service=47ms status=500 bytes=512 protocol=https
2024-03-07T01:40:16.132901+00:00 app[web.1]: 10.1.61.154 - - [07/Mar/2024:01:40:16 +0000] "POST /api/dandisets/000029/versions/draft/assets/ HTTP/1.1" 500 145 "-" "dandi/0.60.0+11.gee18a15e requests/2.31.0 CPython/3.11.8"
2024-03-07T01:40:17.783794+00:00 app[web.1]: 10.1.61.154 - - [07/Mar/2024:01:40:17 +0000] "POST /api/dandisets/000029/versions/draft/assets/ HTTP/1.1" 500 145 "-" "dandi/0.60.0+11.gee18a15e requests/2.31.0 CPython/3.11.8"
2024-03-07T01:40:17.783956+00:00 heroku[router]: at=info method=POST path="/api/dandisets/000029/versions/draft/assets/" host=api.dandiarchive.org request_id=f4646ed1-bd1e-4967-b18e-3a267cf8b4e3 fwd="73.218.136.210" dyno=web.1 connect=0ms service=38ms status=500 bytes=512 protocol=https
2024-03-07T01:40:19.934664+00:00 heroku[router]: at=info method=POST path="/api/dandisets/000029/versions/draft/assets/" host=api.dandiarchive.org request_id=bf87983c-40ba-41d9-ad2e-7ea1e5c53730 fwd="73.218.136.210" dyno=web.1 connect=0ms service=153ms status=500 bytes=512 protocol=https
2024-03-07T01:40:19.934340+00:00 app[web.1]: 10.1.61.154 - - [07/Mar/2024:01:40:19 +0000] "POST /api/dandisets/000029/versions/draft/assets/ HTTP/1.1" 500 145 "-" "dandi/0.60.0+11.gee18a15e requests/2.31.0 CPython/3.11.8"
2024-03-07T01:40:22.552730+00:00 app[web.1]: 10.1.61.154 - - [07/Mar/2024:01:40:22 +0000] "POST /api/dandisets/000029/versions/draft/assets/ HTTP/1.1" 500 145 "-" "dandi/0.60.0+11.gee18a15e requests/2.31.0 CPython/3.11.8"
2024-03-07T01:40:22.552988+00:00 heroku[router]: at=info method=POST path="/api/dandisets/000029/versions/draft/assets/" host=api.dandiarchive.org request_id=bf4b5372-c419-4939-aed2-682773abd707 fwd="73.218.136.210" dyno=web.1 connect=0ms service=48ms status=500 bytes=512 protocol=https
2024-03-07T01:40:25.712900+00:00 app[web.1]: 10.1.61.154 - - [07/Mar/2024:01:40:25 +0000] "POST /api/dandisets/000029/versions/draft/assets/ HTTP/1.1" 500 145 "-" "dandi/0.60.0+11.gee18a15e requests/2.31.0 CPython/3.11.8"
2024-03-07T01:40:25.713061+00:00 heroku[router]: at=info method=POST path="/api/dandisets/000029/versions/draft/assets/" host=api.dandiarchive.org request_id=690a9bbf-d31d-41f5-9d09-38335a3620de fwd="73.218.136.210" dyno=web.1 connect=0ms service=62ms status=500 bytes=512 protocol=https
I guess should be some kind of 400 code
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the API POST /api/dandisets/{id}/versions/draft/assets/ endpoint and trace how an asset path is validated. Reproduce the 596-character upload from the issue, then verify that an over-limit path returns a client error instead of 500 responses and repeated retries; the client-side call sites are dandi/upload.py and dandi/files/bases.py.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- django, python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100