dandi / dandi/dandi-archive

Add check/handle gracefully attempt to upload/rename asset to have path longer than 512 characters

Open
#1,889 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug UX
Dominant language
Python
Stars
26
Forks
21
Avg merge
4d 23h
Merged PRs (30d)
15

Description

512 is current length limit for the path, and it is more than enough. (the longest path in archive I found is 192:

dandi@drogon:/mnt/backup/dandi/dandisets$ declare m=0;for ds in 00*; do git -C $ds ls-tree -r --name-only HEAD; done | awk '{ if (length($0) > max) max = length($0) } END { print max }'
192
dandi@drogon:/mnt/backup/dandi/dandisets$ pwd
/mnt/backup/dandi/dandisets

but if I do attempt to upload a file with path length greater -- it causes 500

❯ DANDI_DEVEL=1 dandi upload --allow-any-path $fn
2024-03-06 20:34:17,039 [    INFO] Found 2 files to consider
PATH                              SIZE    ERRORS PROGRESS STATUS          MESSAGE        
dandiset.yaml                     3.0 kB                  skipped         should be ed...
...901234567890123456789_data.dat 4 Bytes   0             producing asset                
Summary:                          3.1 kB                  1 skipped       1 should be ...
                                                          1 producing ...                
...901234567890123456789_data.dat 4 Bytes   0             ERROR           500 Server E...
Summary:                          3.1 kB                  1 skipped       1 should be ...
                                                          1 ERROR         1 500 Server...
2024-03-06 20:35:13,412 [    INFO] Logs saved in /home/yoh/.local/state/dandi-cli/log/20240307013414Z-313889.log
Traceback (most recent call last):
  File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/bin/dandi", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/click/decorators.py", line 45, in new_func
    return f(get_current_context().obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/dandi/dandi-cli-master/dandi/cli/base.py", line 126, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/dandi/dandi-cli-master/dandi/cli/cmd_upload.py", line 103, in upload
    upload(
  File "/home/yoh/proj/dandi/dandi-cli-master/dandi/upload.py", line 459, in upload
    raise upload_err
  File "/home/yoh/proj/dandi/dandi-cli-master/dandi/upload.py", line 357, in process_path
    for r in dfile.iter_upload(
  File "/home/yoh/proj/dandi/dandi-cli-master/dandi/files/bases.py", line 455, in iter_upload
    r = client.post(
        ^^^^^^^^^^^^
  File "/home/yoh/proj/dandi/dandi-cli-master/dandi/dandiapi.py", line 306, in post
    return self.request("POST", path, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/dandi/dandi-cli-master/dandi/dandiapi.py", line 200, in request
    for i, attempt in enumerate(
  File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/tenacity/__init__.py", line 347, in __iter__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/tenacity/__init__.py", line 325, in iter
    raise retry_exc.reraise()
          ^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/tenacity/__init__.py", line 158, in reraise
    raise self.last_attempt.result()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/yoh/proj/dandi/dandi-cli-master/dandi/dandiapi.py", line 236, in request
    result.raise_for_status()
  File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.11/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.dandiarchive.org/api/dandisets/000029/versions/draft/assets/
DANDI_DEVEL=1 dandi upload --allow-any-path $fn  2.27s user 2.93s system 8% cpu 58.988 total
❯ 
❯ echo $fn
sub-123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789/sub-123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789_ses-1/sub-123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789_data.dat
❯ echo ${#fn}
596

and as we on 500 re-try -- poor dandi-archive keeps 500ing:

2024-03-07T01:40:14.785006+00:00 app[web.1]: 10.1.61.154 - - [07/Mar/2024:01:40:14 +0000] "POST /api/dandisets/000029/versions/draft/assets/ HTTP/1.1" 500 145 "-" "dandi/0.60.0+11.gee18a15e requests/2.31.0 CPython/3.11.8"
2024-03-07T01:40:14.785346+00:00 heroku[router]: at=info method=POST path="/api/dandisets/000029/versions/draft/assets/" host=api.dandiarchive.org request_id=4a509b4c-8716-4350-973c-33deaa1761f1 fwd="73.218.136.210" dyno=web.1 connect=0ms service=48ms status=500 bytes=512 protocol=https
2024-03-07T01:40:16.133195+00:00 heroku[router]: at=info method=POST path="/api/dandisets/000029/versions/draft/assets/" host=api.dandiarchive.org request_id=812d665f-2909-42de-88f3-8c400639f8fc fwd="73.218.136.210" dyno=web.1 connect=0ms service=47ms status=500 bytes=512 protocol=https
2024-03-07T01:40:16.132901+00:00 app[web.1]: 10.1.61.154 - - [07/Mar/2024:01:40:16 +0000] "POST /api/dandisets/000029/versions/draft/assets/ HTTP/1.1" 500 145 "-" "dandi/0.60.0+11.gee18a15e requests/2.31.0 CPython/3.11.8"
2024-03-07T01:40:17.783794+00:00 app[web.1]: 10.1.61.154 - - [07/Mar/2024:01:40:17 +0000] "POST /api/dandisets/000029/versions/draft/assets/ HTTP/1.1" 500 145 "-" "dandi/0.60.0+11.gee18a15e requests/2.31.0 CPython/3.11.8"
2024-03-07T01:40:17.783956+00:00 heroku[router]: at=info method=POST path="/api/dandisets/000029/versions/draft/assets/" host=api.dandiarchive.org request_id=f4646ed1-bd1e-4967-b18e-3a267cf8b4e3 fwd="73.218.136.210" dyno=web.1 connect=0ms service=38ms status=500 bytes=512 protocol=https
2024-03-07T01:40:19.934664+00:00 heroku[router]: at=info method=POST path="/api/dandisets/000029/versions/draft/assets/" host=api.dandiarchive.org request_id=bf87983c-40ba-41d9-ad2e-7ea1e5c53730 fwd="73.218.136.210" dyno=web.1 connect=0ms service=153ms status=500 bytes=512 protocol=https
2024-03-07T01:40:19.934340+00:00 app[web.1]: 10.1.61.154 - - [07/Mar/2024:01:40:19 +0000] "POST /api/dandisets/000029/versions/draft/assets/ HTTP/1.1" 500 145 "-" "dandi/0.60.0+11.gee18a15e requests/2.31.0 CPython/3.11.8"
2024-03-07T01:40:22.552730+00:00 app[web.1]: 10.1.61.154 - - [07/Mar/2024:01:40:22 +0000] "POST /api/dandisets/000029/versions/draft/assets/ HTTP/1.1" 500 145 "-" "dandi/0.60.0+11.gee18a15e requests/2.31.0 CPython/3.11.8"
2024-03-07T01:40:22.552988+00:00 heroku[router]: at=info method=POST path="/api/dandisets/000029/versions/draft/assets/" host=api.dandiarchive.org request_id=bf4b5372-c419-4939-aed2-682773abd707 fwd="73.218.136.210" dyno=web.1 connect=0ms service=48ms status=500 bytes=512 protocol=https
2024-03-07T01:40:25.712900+00:00 app[web.1]: 10.1.61.154 - - [07/Mar/2024:01:40:25 +0000] "POST /api/dandisets/000029/versions/draft/assets/ HTTP/1.1" 500 145 "-" "dandi/0.60.0+11.gee18a15e requests/2.31.0 CPython/3.11.8"
2024-03-07T01:40:25.713061+00:00 heroku[router]: at=info method=POST path="/api/dandisets/000029/versions/draft/assets/" host=api.dandiarchive.org request_id=690a9bbf-d31d-41f5-9d09-38335a3620de fwd="73.218.136.210" dyno=web.1 connect=0ms service=62ms status=500 bytes=512 protocol=https

I guess should be some kind of 400 code

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the API POST /api/dandisets/{id}/versions/draft/assets/ endpoint and trace how an asset path is validated. Reproduce the 596-character upload from the issue, then verify that an over-limit path returns a client error instead of 500 responses and repeated retries; the client-side call sites are dandi/upload.py and dandi/files/bases.py.

Written by the indexing model from the issue text.

Assessment

Tech stack
django, python
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.