download_url gives no retry and a confusing, redundantly-nested error on large-file IncompleteRead/connection-reset failures

Start at the datalad.api.download_url() entry point and the datalad download-url command, then follow the download stack shown in the traceback, including download_url.py, base.py, http.py, and response.py. Reproduce a large transfer failure if possible and determine how retries or resumption should apply. Done means transient failures are handled or clearly surfaced with diagnostics, and redundant exception rendering is reduced.

Filed here for now; to be transferred to datalad/datalad.

Summary

Downloading a large (~6.4 GiB) file via datalad.api.download_url() (also reachable via datalad download-url) failed twice, both times with IncompleteRead/ProtocolError wrapped in DownloadError, raised all the way up as an uncaught IncompleteResultsError with no retry attempted. Filing this from ReproNim/containers, where our create_singularities script uses download_url() to fetch NeuroDesk .simg images and had a run killed outright by this.

Environment

• datalad: <fill in: datalad --version>
• git-annex: <fill in: git annex version>
• OS: Debian/GNU Linux
• URL involved: https://neurocontainers.neurodesk.workers.dev/openmsk_0.2.0_20260908.simg (served from a Cloudflare Worker)

What happened

Two separate runs, days apart, both failed partway through the same ~6.4 GiB (6,859,161,600 byte) download, at different offsets each time:

download_url(error): .../neurodesk-openmsk--0.2.0.simg (file) [DownloadError(ProtocolError(('Connection broken: IncompleteRead(851223088 bytes read, 6007938512 more expected)', IncompleteRead(851223088 bytes read, 6007938512 more expected))) -caused by- IncompleteRead(851223088 bytes read, 6007938512 more expected)) -caused by- ('Connection broken: IncompleteRead(851223088 bytes read, 6007938512 more expected)', IncompleteRead(851223088 bytes read, 6007938512 more expected)) -caused by- IncompleteRead(851223088 bytes read, 6007938512 more expected)] [ProtocolError(('Connection broken: IncompleteRead(851223088 bytes read, 6007938512 more expected)', IncompleteRead(851223088 bytes read, 6007938512 more expected))) -caused by- IncompleteRead(851223088 bytes read, 6007938512 more expected)]

download_url(error): .../neurodesk-openmsk--0.2.0.simg (file) [DownloadError(ProtocolError(('Connection broken: IncompleteRead(5813695055 bytes read, 1045466545 more expected)', IncompleteRead(5813695055 bytes read, 1045466545 more expected))) -caused by- IncompleteRead(5813695055 bytes read, 1045466545 more expected)) -caused by- ('Connection broken: IncompleteRead(5813695055 bytes read, 1045466545 more expected)', IncompleteRead(5813695055 bytes read, 1045466545 more expected)) -caused by- IncompleteRead(5813695055 bytes read, 1045466545 more expected)] [ProtocolError(('Connection broken: IncompleteRead(5813695055 bytes read, 1045466545 more expected)', IncompleteRead(5813695055 bytes read, 1045466545 more expected))) -caused by- IncompleteRead(5813695055 bytes read, 1045466545 more expected)]

Full traceback for the second occurrence:

datalad.support.exceptions.IncompleteResultsError: Command did not complete successfully. 1 failed:
[{'action': 'download_url',
  'error_message': "ProtocolError(('Connection broken: "
                   'IncompleteRead(5813695055 bytes read, 1045466545 more '
                   "expected)', IncompleteRead(5813695055 bytes read, "
                   '1045466545 more expected))) -caused by- '
                   'IncompleteRead(5813695055 bytes read, 1045466545 more '
                   'expected)',
  'exception': ProtocolError(('Connection broken: IncompleteRead(5813695055 bytes read, 1045466545 more expected)', IncompleteRead(5813695055 bytes read, 1045466545 more expected))) -caused by- IncompleteRead(5813695055 bytes read, 1045466545 more expected) -caused by- ('Connection broken: IncompleteRead(5813695055 bytes read, 1045466545 more expected)', IncompleteRead(5813695055 bytes read, 1045466545 more expected)) -caused by- IncompleteRead(5813695055 bytes read, 1045466545 more expected)
[download_url.py:__call__:202,base.py:download:555,base.py:access:180,base.py:_download:526,base.py:_download:499,http.py:download:521,response.py:stream:1267,response.py:read:1155,response.py:_raw_read:1021,contextlib.py:__exit__:158,response.py:_error_catcher:925],
  'exception_traceback': '[download_url.py:__call__:202,base.py:download:555,base.py:access:180,base.py:_download:526,base.py:_download:499,http.py:download:521,response.py:stream:1267,response.py:read:1155,response.py:_raw_read:1021,contextlib.py:__exit__:158,response.py:_error_catcher:925]',
  'message': "DownloadError(ProtocolError(('Connection broken: "
             'IncompleteRead(5813695055 bytes read, 1045466545 more '
             "expected)', IncompleteRead(5813695055 bytes read, 1045466545 "
             'more expected))) -caused by- IncompleteRead(5813695055 bytes '
             "read, 1045466545 more expected)) -caused by- ('Connection "
             'broken: IncompleteRead(5813695055 bytes read, 1045466545 more '
             "expected)', IncompleteRead(5813695055 bytes read, 1045466545 "
             'more expected)) -caused by- IncompleteRead(5813695055 bytes '
             'read, 1045466545 more expected)',
  'path': '/path/to/images/neurodesk/neurodesk-openmsk--0.2.0.simg',
  ...

Analysis / requests

1. No retry. download_url() makes no attempt to retry a transient connection failure on a large download -- a single dropped connection anywhere in a multi-GiB transfer aborts the whole command with IncompleteResultsError, and any caller not wrapping every download_url() call in its own retry loop just crashes. Would it be reasonable for download_url/the HTTP downloader to retry (with backoff, and ideally resuming via Range, or restarting) on this class of error?

2. Same total size, different offsets -- likely disk-space-driven, not server-side truncation. Both failures reported the exact same total expected size (6,859,161,600 bytes) but broke at very different points (851 MB vs. 5.8 GB in). A fixed server-side truncation bug would be expected to break at the same offset each time; the varying offset is more consistent with the destination filesystem running low/out of free space at whatever point it happened to be at on that particular run -- i.e. the write side stalls or fails once space is exhausted, and the remote (a Cloudflare Worker in our case) eventually resets the connection, surfacing as IncompleteRead rather than a clear disk-space error. We're separately checking whether that's what happened on our end, but it's worth having download_url's retry (see point 1) at least surface/log available disk space on the target filesystem when this class of error occurs, so it's diagnosable rather than showing up only as an opaque urllib3 exception chain.

3. Minor: redundant error rendering. Within a single failure, the same IncompleteRead instance is shown 3+ times in the message (once embedded in ProtocolError's own message string, once as ProtocolError.args[1], once again via the -caused by- __cause__-chain walk, then this whole thing repeated again for the DownloadError wrapper, and again across the error_message/exception/message result-dict keys). It first reads as if there were several distinct failures, when it's really one. Might be worth having the -caused by- renderer dedupe/short-circuit when it detects it's about to reprint an exception object it's already shown.