theupdateframework / theupdateframework/python-tuf
Investigate/test fetcher retries
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 304
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 17
Description
I'm expecting urllib3 to retry 3 times on common http failures when using urllib3fetcher
There's an interesting failure in sigstore-probers where this does not seem to happen:
Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/urllib3/response.py", line 754, in _error_catcher
yield
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/urllib3/response.py", line 879, in _raw_read
data = self._fp_read(amt, read1=read1) if not fp_closed else b""
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/urllib3/response.py", line 862, in _fp_read
return self._fp.read(amt) if amt is not None else self._fp.read()
~~~~~~~~~~~~~^^^^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/http/client.py", line 479, in read
s = self.fp.read(amt)
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/socket.py", line 719, in readinto
return self._sock.recv_into(b)
~~~~~~~~~~~~~~~~~~~~^^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/ssl.py", line 1304, in recv_into
return self.read(nbytes, buffer)
~~~~~~~~~^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/ssl.py", line 1138, in read
return self._sslobj.read(len, buffer)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
TimeoutError: The read operation timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.13.4/x64/bin/tuf-on-ci-test-client", line 8, in <module>
sys.exit(client())
~~~~~~^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/click/core.py", line 1161, in __call__
return self.main(*args, **kwargs)
~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/click/core.py", line 1082, in main
rv = self.invoke(ctx)
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/click/core.py", line 1443, in invoke
return ctx.invoke(self.callback, **ctx.params)
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/click/core.py", line 788, in invoke
return __callback(*args, **kwargs)
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/tuf_on_ci/client.py", line 84, in client
updater.refresh()
~~~~~~~~~~~~~~~^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/tuf/ngclient/updater.py", line 170, in refresh
self._load_timestamp()
~~~~~~~~~~~~~~~~~~~~^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/tuf/ngclient/updater.py", line 425, in _load_timestamp
data = self._download_metadata(
Timestamp.type, self.config.timestamp_max_length
)
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/tuf/ngclient/updater.py", line 313, in _download_metadata
return self._fetcher.download_bytes(url, length)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/tuf/ngclient/fetcher.py", line 139, in download_bytes
with self.download_file(url, max_length) as dl_file:
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/contextlib.py", line 141, in __enter__
return next(self.gen)
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/tuf/ngclient/fetcher.py", line 102, in download_file
for chunk in chunks:
^^^^^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/tuf/ngclient/urllib3_fetcher.py", line 105, in _chunks
yield from response.stream(self.chunk_size)
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/urllib3/response.py", line 1066, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/urllib3/response.py", line 955, in read
data = self._raw_read(amt)
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/urllib3/response.py", line 878, in _raw_read
with self._error_catcher():
~~~~~~~~~~~~~~~~~~~^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/contextlib.py", line 162, in __exit__
self.gen.throw(value)
~~~~~~~~~~~~~~^^^^^^^
File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/urllib3/response.py", line 759, in _error_catcher
raise ReadTimeoutError(self._pool, None, "Read timed out.") from e # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='tuf-repo-cdn.sigstore.dev', port=443): Read timed out.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in tuf/ngclient/urllib3_fetcher.py, especially _chunks, and trace how tuf/ngclient/fetcher.py's download_file consumes the response stream. Reproduce the sigstore-probers timeout if possible and inspect urllib3's retry behavior for streaming reads. Done means the expected three retries on common HTTP failures are verified or the missing behavior is clearly identified with focused test coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100